Follow us on:

Nginx health check not working

nginx health check not working These thresholds can be tweaked, though: This work is licensed under a Creative Commons Attribution-ShareAlike 3. Nginx can also retry when the upstream call fails, but there isn’t really a proactive way for Nginx to check the backend health without using Nginx Plus. It consists of two servers that provide a web service to users. Even with a default entry on your webserver you are not actually checking the health of your app, just your webserver which is pretty much the same as checking if the tcp port is open. Action: insert dummy event in Elasticsearch and reads it back. This includes RHEL/Centos 7 and AmazonLinux 1. client_body_temp_path with levels does not work in some cases. Just use a TCP health check until AWS allows for customizing the host header sent for the health check. This guide describes the advantages of load balancing. root@instance-1:~# netstat -anlp | grep 80 tcp 0 0 0. The ability to direct health checks to a specific port is a new feature in NGINX Plus Release 8 . If you want to view version and configure options then use the -V flag as shown. 0 2019/04/17 12:14:20 [emerg] 88#88: invalid number of arguments in "set" directive in /tmp/nginx-cfg161907320:4321 nginx: [emerg] invalid number of arguments in "set See full list on nginx. 0 (nginx-plus-r22) Upstreams are IIS. Combinations of only health checks / only monitoring are built trivially like in the case of normal per-worker upstreams. confで分散するstorageservのホスト・ポート番号を設定 • Health Check – simple checks to validate if a backend is available and if not, then it is automatically removed from the rotation to process requests until it is restored or becomes healthy. patch 4. Probe scale factor. $ nginx -V `try_files` not working with nginx docker image 28th July 2019 angular , docker , nginx I have a simple Angular single-page application that I’m trying to serve with Nginx docker image, version 1. . g. A configuration using HTTP works. 0. 0 0. 0 Unported License. An SSL health check succeeds if the SSL handshake succeeds. To confirm that the server is in a listening state for the health check port, run the netstat -ant command on the server. Here, we will check whether the nginx. NGINX Plus does not proxy client connections to unhealthy servers. The script configures a highly available NGINX Plus environment with an active‑passive pair of nodes acting as primary and backup. Check this box so we and our advertising and social media partners can use cookies on nginx. For example, you can fire an alarm when the nginx. 1:25 check fall 3 rise 2 server smtp2 10. – Pykler Jun 3 '15 at 22:15 The final step is to set up NGINX Plus to use port 9200 for the advanced health check. Nginx, a popular web server software, can be configured as a simple yet powerful load balancer to improve your servers resource availability and efficiency. By default, nginx caches answers using the TTL value of a response. Not sure if there is mismatch between healthmonitor and my nginx configurations. Now you only need to a bit to check for HTTPS and redirect if it’s not. First off, I'll explain my situation to you. uk i got that varnish not working . If it's not listening on the configured port, check your web server settings. Upgrade to Pro — share decks privately, control downloads, hide ads and more … Clients send UDP requests to NGINX or NGINX Plus, which monitors the health and availability of UDP servers and does not send requests to failed or overloaded servers. nginx rewrite wordpress. Whenever we are dealing with physical/Legacy apps those may require extra startup time This project uses nginx as caching proxies. 0:80 0. If you add a TLS listener to your Network Load Balancer, we perform a listener connectivity test. Expected response code : [200]. CSS. Nginx has a health check feature in Nginx Plus, but that isn’t an option for me. upstream. nginx health check for UDP: defect nginx does not compile with LibreSSL 2. Check the source code of the blank/white page. sudo docker run --name=healthcheck-demo -d --health-cmd='stat /etc/nginx/nginx. A similar approach to what we did in the Dockerfile can be done to add a health check to the compose file. This is working as expected. 1. Our nginx has its health-check on /healthz. The ngx_http_upstream_hc_module module allows enabling periodic health checks of the servers in a group referenced in the surrounding location. i have two servers in my upstream, when sabotaging the health for one server i see in the status view of healthcheck that the server is down(1), but if i go to the website i'm checking i still come out on it and see a broken page. If you are running HAProxy, a healthcheck script on each MySQL server in the load balancing set should be able to return an HTTP response status. 13. There's not a huge amount of code here, but I have a lot of control over how the check runs. 1. namespace. If false, NGINX ignores incoming X-Forwarded-* headers, filling them with the request information it sees. AWS ELB health check when using nginx and TCP. In this case, the same information will be shown if you use this command. It does exactly what I want it to, no fancy monitoring, no weights, no fancy route check. By default, NGINX and GitLab will log the IP address of the connected client. com NGINX Plus health checks have a slow‑start feature where you can tell NGINX Plus to ramp the load up slowly so servers doesn’t get hammered when they first come back to life. Kubernetes provides a health checking mechanism to verify if a container in a pod is working or not working. The above command simply displays the version number. 8, I used to have Haproxy -> NGinx caching layer -> Apache. For more information please check our reference documentation. 3 15896 3200 ? S Aug25 0:13 nginx : worker process www-data 778 0. An HTTP/HTTPS health check succeeds if the instance returns a 200 response code within the health check interval. CSS. If your GitLab is behind a reverse proxy, you may not want the IP address of the proxy to show up as the client address. This is indicating your container has started, but no health checks have been performed yet. FROM nginx RUN apt-get update && apt-get install -y curl \ && rm -rf /var/lib/apt/lists/* COPY nginx. So, even if the local system is working fine, it might not actually be capable of serving clients, in which case I want the health check to fail. txt Writing a Custom Health Check in Nodejs When a health check command is specified, it tells Docker how to test the container to see if it's working. com See full list on nginx. When creating the ALB, there is a Advanced health check settings dropdown. The health check fails and then the backend service gets no traffic even though everything is working as designed. 3. Most of the content is immediately available most of the time, but some of the time, nginx has to consult other systems over the network. App Engine health check not working. Up until HAProxy 1. conf EXPOSE 443 WORKDIR /etc/nginx # Set the default command to execute # when creating a new container CMD ["nginx", "-g", "daemon off;"] Here is the nginx. For HTTP or HTTPS health check requests, the host header contains the IP address of the load balancer node and the listener port, not the IP address of the target and the health check port. Many assume it is not possible to install Nginx on Windows. i have problem i configure everything like your post, but still varnish is not working , my website load perfect but when i check varnish through isvarnishworking. In your server. This module was available as part of our commercial subscription until 1. Small memory consumption: the memory consumption of processing large concurrent requests is very small. When calling anything on a path that does NOT start with /healthz we either get a response from the service bound to that ingress address, or we get a 404 response from our default http backend. Once a health check fails, the server gets marked as down. service nginx status. The server is marked as unhealthy, and NGINX Plus does not send client requests to it until it once again passes a health check. Under 30000 concurrent connections, only 150m memory is consumed by 10 nginx processes opened (15m * 10 = 150m) Built in health check function: if a web server on the back end of nginx proxy goes down, the front-end access will not be affected NGINX http Health Checks? NGINX http Health Checks? Download the components listed above. Run the nginx-ha-setup script on both nodes as the root user (the script is distributed in the nginx-ha-keepalived package, which must be installed in addition to the base NGINX Plus package). The fail_timeout value controls the time till Active monitoring supported by nginx + in this get check earlier that health check will be going on in the backend in 5-sec interval if it gets fail for one defined time then it will declare Standard “TCP Connect” health check. health_checks. A workaround for this is to run NGINX from a Docker In your NGINX site configuration file (typically in the /etc/nginx/sites-available/ folder), add a single server block listening on port 80 since all traffic is now flowing through that port. Learn about some of the techniques that NGINX users employ to ensure that server failures are detected and worked around, so that you too can build large-scale, highly-available web services. We have to explicitly enable this setting in Nginx so it does keepalive connections to the upstream it’s connecting to. 0. net For this example, consider we have a web application which requires nginx as its web server. Additionally, I recommend max_fails and fail_timeout to define health checks. What I know point of using load balancing/reverse proxy is to improve performance and reliability by distributing the workload across multiple servers. Double check your baseurl. At the moment we have to manually change the health check port after deployment, then everything works as expected. patch according to the dev's instructions using: patch -p0 < /usr/src/nginx_upstream_check_module-master/check_1. Based on the configuration above, NGINX marks a server as down if it fails to respond or responds with an error three times in 30 seconds. - My nginx has enabled TLS but does not support http - APIGEE target server is configured as one-way TLS enabled but do not include truststore so APIGEE will not validate backend server cert. /bin/systemctl status nginx. 0 0. we are using nginx on ubuntu machine and nginx on aws Nginx PPT Link: https://www. 2. com to better tailor ads to your interests. Also, Varnish requires the origin server receiving the health check requests to close the connection for each request. It work fine with dynamic/static I am trying to check the contents of an html file on upstream servers. httpHealthCheck') 0 Kubernetes, Loadbalancing, and Nginx Ingress - AKS NGINX Plus sends special health check requests to each upstream server and checks for a response that satisfies certain conditions. For example, instead of waiting for an actual TCP request from a DNS client to fail before marking the DNS server as down (as in passive health checks), NGINX Plus will send special health check requests to each upstream server and check for a response that If port is not specified, the port 53 is used. . Note: Consul has sophisticated distributed health checks, so the additional NGINX Plus health_check is not necessarily needed, depending on the configuration of Consul's health checks and the update interval NGINX Plus uses to S Aug25 1:38 nginx: worker process www-data 777 0. It was superseded by the ngx_http_api_module module in 1. If a server is marked down, NGINX continues to probe every 30 seconds. These Google Kubernetes Engine ( shortly GKE ) creates a default health check to verify the state of the backend services and in the case that they are unhealthy it will not allow any client to connect to it by returning a 502 response. 0:* LISTEN 10325/varnishd The controls are not as fine grained as in NGinx, but it's fine for when you just need a small object caching layer. I have a number of nginx doing load balancing sitting behind AWS's network load balancers (TCP) - which seem to only support TCP checks. The ngx_http_status_module module provides access to various status information. Do you mean point the health checks at say '/elb-status' and the add the above server block? is that it? does the /elb-status url need to exist? thx again – Adam Jun 25 '13 at 0:52 worked perfectly when I put /elb-status in the ELB and added the server block above - thankyou so much!!! greatly apprecated – Adam Jun 25 '13 at 0:58 Official command to check Nginx running or not. Active UDP Health Checks. A TCP health check succeeds if the TCP connection succeeds. service nginx status. The following articles describe load balancing with NGINX Plus in more detail: With the port parameter to the health_check directive, we instruct NGINX not to connect to the regular IMAP port [when sending the health check], but rather to a different port [here, 8080]. HTTPS not working when putting ssl certificate with nginx in docker 21st February 2021 docker , docker-compose , https , nginx , ssl I want to pass my server in HTTPS with docker nginx image, but it doesn’t work, I have create a SSL certificate with openSSL like this on my server: The application is listening on the health check port. The health check feature is pretty cool and it’s also pretty smart. In your nginx config make sure your proxy_set_headers and sch are set, location / {proxy_set_header X-Forwarded-Host $host; The Nginx ingress controller, when used in this mode, does not create any AWS resources and is not visible to the internet. • Exclusive to NGINX Plus 34. NGINX Open Source Plugin Installation. 0. If proxy_responses is not default, proxy_timeout expiration is considered a UDP In the Name field, type a human-readable identifier for the health check (e. Not understading how NGINX Reverse Proxy handle image/video file. 1, an upgrade to the HTTP protocol. It is as easy as adding the directive health_check in the location context, as long as there is a proxy_pass directive that specifies an upstream server: Learn how to configure Nginx as HTTP load balancer. nginx version: nginx/1. , West Coast Origin Check). 6; Filename, size File type Python version Upload date Hashes; Filename, size nginx-amplify-agent-health-check-0. This deactivation will work even if you later click Accept or submit a form. 7 RUN apt-get update && apt-get install -y wget HEALTHCHECK CMD wget -q --method=HEAD localhost/system-status. Release 1. Name servers are queried in a round-robin fashion. Thanks. From the Request menu, select an HTTP verb. To learn more about working with Nginx, we recommend going over the following tutorials. With no health check specified, Docker has no way of knowing whether or not the services running within your container are actually up or not. conf /etc/nginx/nginx. However, using the same check with HTTPS does not work. HAProxy does have health check support, in fact, it supports TCP and HTTP health checks. In particular, make sure that proxyname and proxyport are set. Nginx is a web server that is very popular with Linux and BSD systems. In this case, the primary server’s keepalived daemon will enter the “fault” state. 0. I set a timeout for the request call, I check the HTTP status code of the response, and I write log entries on success or failure (which get recorded by Docker and you can see them in docker container inspect ). Nginx upstream health check – custom header. 2. Keepalive connections are only supported as of HTTP/1. 11. If a health check fails, the server will be considered unhealthy. In the match block, I’m defining the request NGINX sends and the specific response it expects. Otherwise, refer this post to install nginx-full. Contact your cloud provider for more support on the issue. I have since been able to remove all the NGinx servers. How does nginx detect if it is a failed > >> connection and retry next? There are two types of such errors. Check your OS firewall settings to make sure that incoming traffic to the port is allowed. . In these settings you can override the health check port. I'm new to Nginx. Checking the logs in the Nginx container with the command kubectl logs POD_NAME -c nginx (hint: tab completion works for the pod names, you do not have to type the whole thing), the health check entries are quite obvious. This results in socket read error. HAProxy does have health check support, in fact, it supports TCP and HTTP health checks. Nginx may reject these requests. here: Health checks – Exchange includes a health‑check mechanism for several applications that integrates easily with NGINX Plus health checks. E0417 12:14:20. The issue is that the backed is not able to route requests without hostname provided. Hence, this is treated as a failure. So, we need to define 2 services: app and nginx. 0. peers. listen smtpcluster1 0. You can update the health check configuration for your load balancer at New connections can bypass this state and move directly to Reading, most commonly when using “accept filter” or “deferred accept”, in which case NGINX does not receive notice of work until it has enough data to begin working on the response. Nginx is not database-aware, so some additional steps are required to configure the health checks for Galera Cluster backends so that they return something understandable. (For my use case, won't work for everyone) NGINX Ingress Controller now offers enhanced TCP/UDP load balancing with support for snippets, health checks, and multiple TransportServer resources. 0. HTTP status code mismatch > >> UDP which is connectionless. If you have configured NGINX with root /path/to/magento/pub (which you should, as it's more secure), then you need to edit the health check VCL code in order to remove pub, e. 0. Test Consul's health checks; Once you have completed this tutorial you will end up with an architecture like the one diagramed below. 1. Active 1 year, 3 months ago. It also uses the health_check instruction to enable advanced performance monitoring: by default, NGINX Plus sends a “/” request backend to each server in the group every five seconds. What I researched so far is when I switch to Cluster (health checks work then), I will loose the client IP and maybe get an additional hop inside the cluster and also the LB maybe forwards traffic to a node with less or no pods, but the ingress will route correctly. php. 1. 13. Patch nginx-sticky-module-ng with: patch -p1 < patches/upstream_check. -- Sergey Osokin On Thu, Oct 01, 2020 at 02:13:54PM -0400, Jeff Creek wrote: > I am trying to check the contents of an html file on upstream servers. 17. 8. If true, NGINX passes the incoming X-Forwarded-* headers to upstreams. To check the version of Nginx web server installed on your Linux system, run the following command. 9. Last but not least, application load balancing, application health checks, activity monitoring and on-the-fly reconfiguration of server groups are available as part of our paid NGINX Plus subscriptions. Received response code : 404 The warning message above states that the expected response code for the health check API was 200, but the actual response received is 404. unhealthy goes above certain threshold for the production kubernetes. 0: If the origin server does not receive a host it expects, it may issue a 301 or 302 redirect causing the health check to fail. The health check services become shared, the monitoring service gets replaced with a simple content handler reportPeers because peers are now shared between all Nginx worker processes. xml file make sure you are using the one for HTTPS - Proxying Jira via Apache or Nginx over HTTPS. You can use the Health Check plugin to determine which plugin or theme is causing this. • Monitoring –simple statistics relating to NGINX are available for monitoring purposes but are limited. To simulate a TCP connection between the load balancer and backend, run the telnet <backend An Nginx configuration however, by default, only talks HTTP/1 to the upstream (in this exampple a Varnish). 2 kB) File type Source Python version None Upload date May 30, 2018 Nginx has a health check feature in Nginx Plus, but that isn’t an option for me. You can check the status of nginx using the following command: $ sudo systemctl status nginx Step: 6 To install HAProxy on Debian 10 (Buster), run the following command on the load-balancer. Since this is related to the commercial version of NGINX - NGINX Plus, I'd recommend to raise a request with 24x7 NGINX Support Team through the My F5 Portal. 3 This guide covers basic commands and practices for managing an Nginx server, including how to start, stop, and check the status of Nginx, how to find a website’s document root, and how to check the syntax of an Nginx configuration file. If looking up of IPv6 addresses is not desired, the ipv6=off parameter can be specified. NGINX will get a list of healthy servers from Consul service discovery via Consul template and will balance internet traffic to those servers according to its own configuration. 0. A starting container with no health checks executed yet (image by author) With the default configuration for health checks, the health-check command doesn’t get executed immediately, so the initial status you see is (health: starting). This type of health check verifies whether the overall behavior of the system is working, it can detect tricky malfunctions that would otherwise only be visible in real customer transactions. 12. tar. For example, to check every five minutes or so that a web-server is able to serve the site’s main page within three seconds: The final step is to set up NGINX Plus to use port 9200 for the advanced health check. 3 15896 3200 ? S Aug25 1:49 nginx : worker process www-data 779 0. The ability to direct health checks to a specific port is a new feature in NGINX Plus Release 8 . Ingress supports more advanced use cases, such as: It will also instruct NGINX Plus to put a default health_check on all the available servers in the upstream group. Depends on: Nginx, API, Elasticsearch. Use this option when NGINX is behind another L7 proxy / load balancer that is setting these headers. Working with qlmanage (in case of Mac) health check nginx. That is not true at all because the web server can indeed be installed on Windows 10. » Prerequisites The health check returned the warning message: HTTP response code from health monitoring service does not match. go:182] Unexpected failure reloading the backend: ----- Error: exit status 1 2019/04/17 12:14:20 [notice] 88#88: ModSecurity-nginx v1. 522354 8 controller. In Docker, health checks can be specified in the Dockerfile as well as in a compose file. When the Nginx health check on the primary server indicates that Nginx is no longer running. Check the application configuration port to verify that it is running. For example if only check 2 goes down, you can easily see that there is an issue with Elasticsearch, since the other downstream services are also checked in the first health check. It would ping the server and check for 200 status response. 10. If a server fails to send out a response, it is automatically marked as such, and nginx avoids sending requests to it for some time. Based on the logs provided, a backend health check fails in Varnish for /pub/health_check. Default GKE Health check . c. This is sent to each of the servers defined in the A unique advantage of Kong is that both active and passive health checks are offered for free in the Community Edition (CE). After this command you will see something like this. If you are still getting issues, check that you installed SSL certificates properly. 9. If the origin server does not close the connection, health checks will time out and fail. patch 3. 0. The send directive is a hexadecimal representation of a complete MQTT CONNECT packet with a Client identifier (ClientId) of nginx health check. 0. If the source code of the sitemap page is empty, there must be a conflict with another plugin or the theme. For now, we add the "-health-status" parameter to the controller. Update the health check configuration. 17. If you want to check a different path or to expect a different response code, you can use a custom health check path. For example: site bindings in IIS, server block in NGINX and virtual host in Apache. By default, nginx will look up both IPv4 and IPv6 addresses while resolving. Recently a few have stopped working / frozen - they still seem to accept a tcp connection from the NLB - which leads the health check not to fail. However, according to Nginx’s own website, there are a few performance limitations that have not been mitigated so far. Active Health Checks allow testing a wider range of failure types and are available only for NGINX Plus. We will set the Command while running the Docker Container. Closes nginxinc#90 thisfailsThe parameter requires the server to be marked as not performing well (starting with the default value) when three checks fail. 2+. It will notify the secondary server that it should transition to the master state and claim the floating IP. conf file exists or not. I was hoping someone might have an idea here. With this set the DO-Loadbalancer fails Health-Checks. We will set the Command while running the Docker Container. 2+. How does Nginx work? Nginx acts as a single entry point to a distributed web application working on multiple separate servers. Since the DO Load Balancer uses a the private IPs to check the health of each droplet and not a domain, the health checks were failing (Nginx was Each health check has an associated check frequency, inversely proportional to the configured check interval: 1 ⁄ (check interval) When you associate a health check with a backend service, you establish a base frequency used by each prober for backends on that backend service. In order to actively check the health of upstream servers, nginx can send special requests to verify it. 19. Long story short, there was no Default app for the application settings in ServerPilot, and “Deny requests for unknown domains” was ON. The TCP health check was working, but I needed HTTP. 2:25 check fall 3 rise 2 Memcached Cluster When nginx proxies requests to these servers, it also passively performs a health check. 1. g. • For UDP it will send “nginx health check” every 5 seconds and expect the absence of ICMP “Destination Unreachable” in the response. In the Request field, type the path to visit when performing the check. If proxy_responses is not zero, ICMP Destination Unreachable signals a UDP error. Live activity monitoring – NGINX Plus includes a dashboard that provides key load and performance metrics in real time, including TCP metrics in NGINX Plus R6 and later. d. 0 also introduces a WAF policy for easier configuration of NGINX App Protect, Istio compatibility, and more. slideshare. If these steps don’t solve your problems, then you may have the wrong configuration in your load balancer. Closes nginxinc#90 Nginx also provides a passive health check for free. If a connection to the server cannot be established, the health check fails, and the server is considered unhealthy. name For Nginx Plus to work with Sysdig, you first need to expose the status endpoint. Ask Question Asked 1 year, 3 months ago. 0: success - the container is healthy and ready for use 1: unhealthy - the container is not working correctly 2: reserved - do not use this exit code. Files for nginx-amplify-agent-health-check, version 0. Try accessing your instance without the load balancer enabled. Amazon Elastic Load Balancers (ELB) don’t offer passive checks. conf below. If any communication error or timeout occurs (the server responds with a status code outside the range from 200 through 399) the health check fails. The server group must reside in the shared memory. 0 0. Patch nginx_upstream_check_module with the check_1. You can have NGINX look for a different address to use by adding your reverse proxy to the real_ip_trusted_addresses list: Check Nginx Version. The following is an example of a simple web server health check. GKE Ingress with container-native load balancing does not detect health check (Invalid value for field 'resource. $ nginx -v nginx version: nginx/1. You can set the port to something like 8080 and then your server becomes: server { listen 8080; location /elb-status { access_log off; return 200 'A-OK!'; add_header Content-Type text/plain; } } Unfortunately, Open Source Nginx native does not support health checks, for that you need to install a module which is called nginx_upstream_check_module (This module is not distributed with the Nginx load balancer, image source: google Sometimes if your server is available, then sends the request to the server but if its inner functionality may not work properly, at that time your servers You can check that by running following command: nginx -V 2>&1 | grep -o with-http_stub_status_module If you see following output, you are good to go ahead. Nginx offers passive health checks in the community edition, but active health checks are included only in the paid edition, Nginx Plus. service. Note: this plugin does not work on Linux distributions that use older versions of libstdc++. Use this option if NGINX is exposed directly to the internet, or it's behind a L3/packet-based load balancer that doesn't alter the source IP in the packets. Ask questions Kong Upstream health Check is not working for https protocol Summary We are trying to configure healthcheck for Kong Upstream, it works with http protocol, but fails for https protocol. Nginx can also retry when the upstream call fails, but there isn’t really a proactive way for Nginx to check the backend health without using Nginx Plus. If it is present, the default server listening on port 80 with the health check location "/nginx-health" gets added to the main nginx configuration. Mail Server Health Checks. We almost have this working, we set BE protocol to "ssl" and the listeners are configured in a way that works, but it makes the ELB health check use SSL (fine) against the non-SSL node port (not fine). FROM nginx:1. 0. We can add a Compose configuration parameter to our nginx service to define the check process: By default, Ingress performs a periodic health check by making a GET request on the / path to determine the health of the application, and expects a HTTP 200 response. 0:8080 mode tcp server smtp1 10. Use a unique path. By default, nginx will give a backend node 10 seconds to respond to a user’s request and a single failure to do so will result in the node being marked as unavailable, with nginx checking again every 10 seconds to see if it’s back up. Hi there, i'm trying to use the HttpHealthcheckModule for nginx, but i have some troubles with it. conf || exit 1' healthcheck-demo /* This line must be without the breaks, it's done for viewing purpose*/ The other way is to health check the container, like nginx, which helps to check the status by serving a file in it. Is there a way to setup Nginx plus to pass also Host header while execution a health check? It looks like nginx resolves an IP address of the server and then using it instead of hostname specified. 6. 0. It just sits there functioning as a reverse proxy, passing requests from The instructions below use NGINX from the official Linux repositories and pre-built binaries for the plugins. gz (10. how to configure load balancer in linux. nginx health check not working