# Metrics and dashboards
Sourcegraph ships with [Grafana](https://grafana.com) for dashboards, [Prometheus](https://prometheus.io/) for metrics and alerting. We also provide [built-in alerting](/self-hosted/observability/alerting) for these metrics.
## Grafana
Site admins can view the Grafana monitoring dashboards on a Sourcegraph instance:
1. Go to **User menu > Site admin**.
1. Open the **Monitoring** page from the link in the left sidebar. The URL is `https://sourcegraph.example.com/-/debug/grafana/`.
### Dashboards
A complete [dashboard reference](/self-hosted/observability/dashboards) is available for more context on our available service dashboards and panels.
Additional dashboards can also be set up—see [Grafana configuration](#grafana-configuration) for more details.
#### View documentation
On service dashboards, each metric panel has links attached that lead to relevant [alert solutions](/self-hosted/observability/alerts) or [panel documentation](/self-hosted/observability/dashboards).
These can be accessed from the top left corner of each panel.
#### View alerts
The Overview dashboard includes a high-level summary of alert events across all Sourcegraph services.
On service dashboards, a summary of alert events for that service is available.
Alerts can be filtered by level using the "Filter alert level" selector at the top of the dashboard.
This information can be overlaid on all panels in that dashboard by enabling the "Alert events" toggle.
#### View version change events
On service dashboards, annotations can be displayed for occasions when a Sourcegraph version change is detected by enabling the "Version changes" toggle at the top of the dashboard.
This can be useful for correlating metrics and alerts with Sourcegraph deployment and upgrade events.
#### Querying metrics
Specific metrics can be queried using Grafana's Explore panel, available at `/-/debug/grafana/explore` on your Sourcegraph instance.
The query for each individual panel can be viewed and explored by clicking on the panel and using the "Explore" option.
### Grafana configuration
Sourcegraph deploys a customized image of Grafana, which ships with Sourcegraph-specific dashboard definitions.
To provide custom dashboards, a directory containing dashboard JSON specifications can be mounted in the Docker container at `/sg_grafana_additional_dashboards`.
Changes to files in that directory will be detected automatically while Grafana is running.
More behavior can be controlled with [environmental variables](https://grafana.com/docs/grafana/latest/administration/configuration/#configure-with-environment-variables).
> As a workaround, site admins can [connect to Grafana directly](#accessing-grafana-directly) to make changes using the Grafana UI.
### Accessing Grafana directly
For most use cases, you can access Grafana [through your Sourcegraph instance](#grafana).
Follow the instructions below to access Grafana directly to, for example, edit configuration directly.
> NOTE: Most of the dashboards that Sourcegraph ships with are not configurable through the Grafana UI.
> In general, we recommend [these configuration methods instead](#grafana-configuration).
If you are using the [Kubernetes deployment option](/self-hosted/deploy/kubernetes/), you can access Grafana directly using Kubernetes port forwarding to your local machine:
```sh
kubectl port-forward svc/grafana 3370:30070
```
Grafana will be available http://localhost:3370/-/debug/grafana.
If you are using the [Docker Compose deployment option](/self-hosted/deploy/), Grafana is available locally at http://localhost:3370/-/debug/grafana without any additional setup.
> **Note:** The [Docker Single Container](/self-hosted/deploy/docker-single-container/) deployment mode has been removed in 7.0.0. Please [migrate to Docker Compose](/self-hosted/deploy/docker-compose/migrate).
If Sourcegraph is deployed to a remote server, then access via an SSH tunnel using a tool such as [sshuttle](https://github.com/sshuttle/sshuttle) is required to establish a secure connection to Grafana.
To access the remote server using `sshuttle` from your local machine:
```bash
sshuttle -r user@host 0/0
```
Grafana will be available at http://host:3370/-/debug/grafana.
### Grafana security
WARNING: By default, our Grafana container runs in anonymous mode with authentication disabled, relying on Sourcegraph's authentication and authorization when accessed through your Sourcegraph instance.
We recommend you use your network security controls to prevent access to Grafana's listening port, or enable Grafana's built-in authentication.
To enable Grafana's built-in authentication, add `GF_AUTH_ANONYMOUS_ENABLED` as `false` in the Grafana container's environment variables in your deployment override file, then redeploy the Grafana container.
We also recommend that you customize the default admin username and password.
For new deployments, configure the `GF_SECURITY_ADMIN_USER` and `GF_SECURITY_ADMIN_PASSWORD` environment variables on the Grafana container, using your deployment pipeline's secrets management tool, before initial deployment, to initialize the default admin user on first deployment.
For existing deployments, you can either log in to the Grafana web UI with the default credentials and it will prompt you to change the password, or you can reinitialize Grafana:
1. Export any customized dashboards via JSON (if applicable)
2. Configure the environment variables the same as a new instance
3. Delete the Grafana container's storage volume
4. Redeploy the Grafana container
5. Import any customized dashboards (if applicable)
The contents of the Grafana container's storage volume are ephemeral, so metrics data is not lost in this process.
```yaml
# Helm override
grafana:
env:
GF_AUTH_ANONYMOUS_ENABLED:
value: 'false'
GF_SECURITY_ADMIN_USER:
value: 'custom-admin-username'
GF_SECURITY_ADMIN_PASSWORD:
value: 'custom-admin-password'
# Docker Compose override
grafana:
environment:
- 'GF_AUTH_ANONYMOUS_ENABLED=false'
- 'GF_SECURITY_ADMIN_USER=custom-admin-username'
- 'GF_SECURITY_ADMIN_PASSWORD=custom-admin-password'
```
## Prometheus
Prometheus is a monitoring tool that collects application- and system-level metrics over time and makes these accessible through a robust query language.
For most use cases, you can query Prometheus through [Grafana](#grafana) using Grafana's Explore panel, available at `/-/debug/grafana/explore` on your Sourcegraph instance, or simply rely on the dashboards we ship.
### Available metrics
#### High-level alerting metrics
Sourcegraph's metrics include a single high-level metric `alert_count` which indicates the number of `level=critical` and `level=warning` alerts each service has fired over time for each Sourcegraph service.
This is the same metric presented on the **Overview** Grafana dashboard.
> NOTE: We provide [built-in alerting](/self-hosted/observability/alerting) for these alerting metrics to help monitor the health of your Sourcegraph instance.
> Refer to our [alert solutions reference](/self-hosted/observability/alerts) for details on specific alerts.
**Description:** The number of alerts each service has fired for a given alert name and severity level.
**Values:** Although the values of `alert_count` are floating-point numbers, only their whole numbers have meaning.
For example, `0.5` and `0.7` indicate no alerts are firing, while `1.2` indicates exactly one alert is firing and `3.0` indicates exactly three alerts firing.
**Labels:**
| Label | Description |
| -------------- | -------------------------------------------------------------------------------- |
| `service_name` | the name of the service that fired the alert |
| `name` | the name of the alert that the service fired |
| `level` | either `critical` or `warning`, as defined [here](/self-hosted/observability/alerting) |
| `description` | a human-readable description of the alert |
#### Complete reference
A complete reference of Sourcegraph's vast set of Prometheus metrics is not yet available. If you are interested in this, please reach out by filing an issue or contacting us at [support@sourcegraph.com](mailto:support@sourcegraph.com).
### Prometheus configuration
Sourcegraph runs a customized image of Prometheus, which packages a standard Prometheus installation together with rules files and target files tailored to Sourcegraph and quality-of-life integrations such as [the ability to configure alerting from the Sourcegraph web application](/self-hosted/observability/alerting/).
A directory can be mounted at `/sg_prometheus_add_ons`. It can contain additional config files of two types:
- rule files which must have the suffix `_rules.yml` in their filename (ie `gitserver_rules.yml`)
- target files which must have the suffix `_targets.yml` in their filename (ie `local_targets.yml`)
[Rule files](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/)
and [target files](https://prometheus.io/docs/guides/file-sd/) must use the latest Prometheus 2.x syntax.
The environment variable `PROMETHEUS_ADDITIONAL_FLAGS` can be used to pass on additional flags to the `prometheus` executable running in the container.
### Accessing Prometheus directly
Most of the time, Sourcegraph site admins will monitor and query key metrics through [Grafana](#grafana), rather than through Prometheus directly.
Grafana also provides the dashboards that monitor the standard metrics that indicate the health of the instance.
Follow the instructions below to access Prometheus directly instead.
If you are using the [Kubernetes deployment option](/self-hosted/deploy/kubernetes/), port-forward the Prometheus service:
```sh
kubectl port-forward svc/prometheus 9090:30090
```
If you are using the [Docker Compose deployment option](/self-hosted/deploy/), you will need to restart the Sourcegraph container
with a flag `--publish 9090:9090` in the `docker run` command.
> **Note:** The [Docker Single Container](/self-hosted/deploy/docker-single-container/) deployment mode has been removed in 7.0.0. Please [migrate to Docker Compose](/self-hosted/deploy/docker-compose/migrate).
Prometheus will be available http://localhost:9090.
## Using a custom monitoring system
Please refer to our FAQ item, ["Can I consume Sourcegraph's metrics in my own monitoring system (Datadog, New Relic, etc.)"](/admin/faq#can-i-consume-sourcegraph-s-metrics-in-my-own-monitoring-system-datadog-new-relic-etc).