Sourcegraph 3.14: Faster repository permissions, excluding forks and archived repositories by default, and Docker Compose deployment

Explore, navigate, and better understand all code, everywhere, faster with Sourcegraph Universal Code Search. Uber, Lyft, Yelp, and other enterprises rely on Sourcegraph to improve developer productivity and manage large-scale code changes.

Deploy or upgrade: Local | AWS | DigitalOcean | Kubernetes cluster

---

Watch the recording of the 3.14 release livestream with Product Manager Christina Forney.

Faster syncing of GitLab and Bitbucket Server repository permissions

Sourcegraph permissions syncing is now more sophisticated to better handle repository permissions at scale. Rather than updating a user’s permissions when they log in and potentially blocking them from seeing search results, Sourcegraph now syncs these permissions opportunistically, as part of a background service. Users are now able to see results from their search queries immediately. Currently, this improvement is supported for GitLab and Bitbucket Server, with GitHub and other code hosts coming soon. See the documentation for more details.

Enable this feature by adding "permissions.backgroundSync": {"enabled": true} in your site configuration settings.

Forks and archived repositories are excluded from search results by default

The default setting for including forks and archived repositories in your search results has changed from including by default to excluding by default. This is because most often, users want to see the active and original repositories in their queries. You can still use archived:yes or fork:yes filters to show results that include archived and forked repositories.

Go to definition in PRs and simplified sign in with the browser extension

It is now easier to sign in with the Sourcegraph browser extension or native integrations - no need to refresh your code host window! Additionally, we've added support for staying on a pull request when a go to definition is defined within the visible context. For native integration users, these improvements come with an upgrade to Sourcegraph 3.14. Browser extension users can get this by updating to the latest release.

Deploy and scale Sourcegraph with Docker Compose

Sourcegraph 3.14 introduces a new docker-compose deployment model. This is useful for organizations that want to scale Sourcegraph, but don’t use Kubernetes. For any customers running a single Docker container and have experienced instability or issues with scaling, Docker Compose is likely the right solution for you!

Campaigns are easier to create and manage and have reduced the load on code hosts

Behind the scenes, campaign changesets are now kept up to date using a heuristic solution that reduces the load placed on the code host. Previously, each changeset was updated during regularly scheduled update processes. The new implementation prioritizes syncing changesets based on when last they changed and is also aware of real-time changes that arrive via webhooks.

The overall experience of generating diffs, creating and reviewing a campaign, and then managing the changes to closure has been improved.

Using the src-cli to generate diffs and create a campaign plan has been improved:

• It is now easier to create a campaign plan for successful patches when other patches fail for some repositories.
• The dependencies on the external diff and unzip tools have been removed. Instead, git is used to produce diffs.
• .gitignore files are now respected when creating patches.
• On macOS, src CLI can now be installed with Homebrew.

Take a look at the improved documentation for usage and examples.

When creating your campaign in Sourcegraph, it is now clearer what state the campaign is in and is easier to differentiate between draft patches and changesets that have been created on the code host. Additionally, you can now filter the changesets to quickly find which ones need follow-up.

Code change management campaigns are in private beta. Watch the campaigns screencasts to see what we have planned, and apply for early access to campaigns for your organization.

IMPORTANT: Required migration for Kubernetes deployments

A manual migration is required for Kubernetes deployments. As is usual, please refer to the documentation for your deployment type for upgrading:

• Kubernetes upgrade documentation
• Docker Compose upgrade documentation
• Single-container sourcegraph/server upgrade documentation

Improved debugging of Kubernetes deployments with profiling and tracing

Kubernetes deployments now also have additional debug / instrumentation information available under Site Admin → Instrumentation. This information will primarily be used in the event that support needs to, for example, request a performance profile for a specific service of Sourcegraph in order to debug an issue you are facing.

Search insights and alerts added to admin monitoring

The frontend dashboard now shows how search is behaving overall and in detail. Additionally, admins can now be proactively notified when critical parts of Sourcegraph are not working as expected.

Alerts are in place for when:

• Hard search errors (both timeouts and general errors) are high.
• Partial search timeouts are high.
• Search 90th and 99th percentile request duration is high.
• Users are being shown an abnormally large amount of search alert user suggestions and no results.
• The internal indexed and unindexed search services are returning bad responses.
• gitserver may be under heavy load due to many concurrent command executions or under-provisioning.

Estimate required resources for your Sourcegraph instance

The new resource estimator helps admins of new and growing Sourcegraph instances determine the right resources and deployment strategy for their custom scenario. CPU and memory needs are calculated for each service based on the number of repositories, users, large monorepos, and engagement rate of your users.

Option to exclude all forks/archives from syncing to Sourcegraph for GitHub

Some organizations do not want to include any forks or archived repositories in their Sourcegraph instance. Admins can now exclude GitHub forks and/or archived repositories from the set of repositories being mirrored in Sourcegraph. The GitHub repository management can be updated with "exclude": [{"forks": true}] or "exclude": [{"archived": true}] to utilize this feature.

Improved performance and coverage for code intelligence

• Greatly improved the performance of the Go LSIF indexer (indexing time reduced by 60% for aws/aws-sdk-go).
• The Go LSIF indexer now provides hovertext for packages and external symbols.
• Added basic code intelligence for Thrift schema files.

Aggregated search data added to Sourcegraph pings

Sourcegraph collects a small amount of high-level and aggregate data from each instance – you can see the complete list of the data we collect, and our ping philosophy and rules. New usage statistics – specifically, aggregate search latencies, aggregated counts of users and searches for search mode types, and aggregated counts of search filter usage – will be sent to Sourcegraph via pings by default. The aggregated event count metrics can be disabled via the site admin flag disableNonCriticalTelemetry.

Experimental smart search bar

In Sourcegraph 3.13, we introduced the smart search bar as an experimental feature. Many improvements have been made, but it will stay marked as experimental as we continue to solicit feedback. If you or your team are interested in trying this feature, add the following setting to your user, organization, or global settings on your Sourcegraph instance: { "experimentalFeatures": { "smartSearchField": true } }. Tweet @sourcegraph or email [email protected] with your feedback.

Changelog

Added

• Site-Admin/Instrumentation is now available in the Kubernetes cluster deployment 8805.
• Extensions can now specify a baseUri in the DocumentFilter when registering providers.
• Admins can now exclude GitHub forks and/or archived repositories from the set of repositories being mirrored in Sourcegraph with the "exclude": [{"forks": true}] or "exclude": [{"archived": true}] GitHub external service configuration. #8974
• Campaign changesets can be filtered by State, Review State and Check State. #8848
• Counts of users of and searches conducted with interactive and plain text search modes will be sent back in pings, aggregated daily, weekly, and monthly.
• Aggregated counts of daily, weekly, and monthly active users of search will be sent back in pings.
• Counts of number of searches conducted using each filter will be sent back in pings, aggregated daily, weekly, and monthly.
• Counts of number of users conducting searches containing each filter will be sent back in pings, aggregated daily, weekly, and monthly.
• Added more entries (Bash, Erlang, Julia, OCaml, Scala) to the list of suggested languages for the lang: filter.
• Permissions background sync is now supported for GitLab and Bitbucket Server via site configuration "permissions.backgroundSync": {"enabled": true}.
• Indexed search exports more prometheus metrics and debug logs to aid debugging performance issues. #9111
• monitoring: the Frontend dashboard now shows in excellent detail how search is behaving overall and at a glance.
• monitoring: added alerts for when hard search errors (both timeouts and general errors) are high.
• monitoring: added alerts for when partial search timeouts are high.
• monitoring: added alerts for when search 90th and 99th percentile request duration is high.
• monitoring: added alerts for when users are being shown an abnormally large amount of search alert user suggestions and no results.
• monitoring: added alerts for when the internal indexed and unindexed search services are returning bad responses.
• monitoring: added alerts for when gitserver may be under heavy load due to many concurrent command executions or under-provisioning.

Changed

• The "automation" feature was renamed to "campaigns".
  • campaigns.readAccess.enabled replaces the deprecated site configuration property automation.readAccess.enabled.
  • The experimental feature flag was not renamed (because it will go away soon) and remains {"experimentalFeatures": {"automation": "enabled"}}.
• The Kubernetes deployment for existing installations requires a migration step when upgrading past commit 821032e2ee45f21f701 or when upgrading to 3.14. New installations starting with the mentioned commit or with 3.14 do not need this migration step.
• Aggregated search latencies (in ms) of search queries are now included in pings.
• The Kubernetes deployment frontend role has added services as a resource to watch/listen/get. This change does not affect the newly-introduced, restricted Kubernetes config files.
• Archived repositories are excluded from search by default. Adding archived:yes includes archived repositories.
• Forked repositories are excluded from search by default. Adding fork:yes includes forked repositories.
• CSRF and session cookies now set SameSite=None when Sourcegraph is running behind HTTPS and SameSite=Lax when Sourcegraph is running behind HTTP in order to comply with a recent IETF proposal. As a side effect, the Sourcegraph browser extension and GitLab/Bitbucket native integrations can only connect to private instances that have HTTPS configured. If your private instance is only running behind HTTP, please configure your instance to use HTTPS in order to continue using these.
• The Bitbucket Server rate limit that Sourcegraph self-imposes has been raised from 120 req/min to 480 req/min to account for Sourcegraph instances that make use of Sourcegraphs' Bitbucket Server repository permissions and campaigns at the same time (which require a larger number of API requests against Bitbucket Server). The new number is based on Sourcegraph consuming roughly 8% the average API request rate of a large customers' Bitbucket Server instance. #9048
• If a single, unambiguous commit SHA is used in a search query (e.g., repo@c98f56) and a search index exists at this commit (i.e., it is the HEAD commit), then the query is searched using the index. Prior to this change, unindexed search was performed for any query containing an @commit specifier.

Fixed

• Zoekt's watchdog ensures the service is down upto 3 times before exiting. The watchdog would misfire on startup on resource constrained systems, with the retries this should make a false positive far less likely. #7867
• A regression in repo-updater was fixed that lead to every repository's git clone being updated every time the list of repositories was synced from the code host. #8501
• The default timeout of indexed search has been increased. Previously indexed search would always return within 3s. This lead to broken behaviour on new instances which had yet to tune resource allocations. #8720
• Bitbucket Server older than 5.13 failed to sync since Sourcegraph 3.12. This was due to us querying for the archived label, but Bitbucket Server 5.13 does not support labels. #8883
• monitoring: firing alerts are now ordered at the top of the list in dashboards by default for better visibility.
• monitoring: fixed an issue where some alerts would fail to report in for the "Total alerts defined" panel in the overview dashboard.

Removed

• The v3.11 migration to merge critical and site configuration has been removed. If you are still making use of the deprecated CRITICAL_CONFIG_FILE, your instance may not start up. See the migration notes for Sourcegraph 3.11 for more information.

The changelog for this and previous releases is available on GitHub.

Thank you

Thank you to the many people who contributed to Sourcegraph since the last release!

• @souldzin
• @philjones
• @brentshermana
• @prayashm
• @analogue
• @kwangil-ha
• @mipearson
• @machbio
• @daxmc99
• @eculver
• @mntky
• @h1nk
• @craffert0
• @zgrimshell
• @Antoine98

Deploy or upgrade: Local | AWS | DigitalOcean | Kubernetes cluster

From the entire Sourcegraph team (@sourcegraph), happy coding!