Sourcegraph 3.13: Interactive search mode, structural search toggle, and campaigns with custom code execution

Quickly explore and better understand all the code everywhere with Sourcegraph Universal Code Search. With Sourcegraph, every company has access to the same kind of tools that Google and Facebook developers use every day.

Deploy or upgrade: Local | AWS | DigitalOcean | Kubernetes cluster

Interactive search mode

Interactive search mode helps users construct queries using UI elements, and is now enabled by default for all users. In this mode, search filters are more discoverable and the query input is simplified to the search query pattern. A dropdown to the left of the search bar allows users to switch between interactive and plain text modes.

The option to use interactive search mode can be disabled by adding { "experimentalFeatures": { "splitSearchModes": false } } in global settings.

Code-aware structural search toggle

Sourcegraph 3.13 adds a UI toggle for structural search, a code-aware search syntax. Structural search was introduced in Sourcegraph 3.11 and was previously enabled using the patternType:structural query parameter. When enabled, the regexp and case sensitivity toggles will be disabled, since the search types work independently. Structural code search lets you match nested expressions and whole code blocks that can be difficult or awkward to match using regular expressions. Read more about structural search and see examples in our recent blog post "Going beyond regular expressions with structural code search".

Campaigns with custom code execution using src-cli

Sourcegraph campaigns now support running arbitrary code over all your repositories using the Sourcegraph CLI. This is helpful when you need to run a series of steps over your code, or run a formatter after making an update.

The src-cli takes your custom action.json file, containing a scopeQuery and series of commands as steps, as input. It then programmatically downloads zips of each repository that match the scopeQuery, applies each command step, and generates a diff with the changes. The resulting diff file can then be sent to Sourcegraph so you can preview the changes and create the campaign.

For example, if I wanted to bump the version of RxJS in several repositories, my action.json using a Docker container could be:

{
  "scopeQuery": "repohasfile:yarn.lock file:^package.json$ archived:no fork:no rxjs",
  "steps": [
    {
      "type": "docker",
      "image": "sourcegraph/rxjs-upgrade:latest"
    }
  ]
}

See our documentation for more details and examples.

Code change management campaigns are in private beta. Watch the campaigns screencasts to see what we have planned, and apply for early access to campaigns for your organization.

Basic code intelligence support reaches 32 languages

With the addition of GraphQL, Groovy, Objective-C, Pascal, Protobufs, Verilog, and VHDL, Sourcegraph basic code intelligence now supports 32 languages. Every Sourcegraph instance comes enabled with basic code intelligence provided by search based heuristics. For fast and precise code intelligence see our LSIF documentation.

LSIF-based precise code intelligence for Dart

The Dart LSIF indexer makes precise code intelligence available for Dart projects. LSIF (Language Server Index Format) is a file format for precomputed code intelligence data that you can upload to your Sourcegraph instance for your projects. Follow the LSIF quickstart guide to get started and see the most up-to-date information on languages with LSIF indexers at lsif.dev.

New content search keyword

We added a new content: search keyword to help you search for patterns that might clash with other parts of the query. For example, if you want to search for file:file in javascript files, you can now use the following query: file:.js content:"file:file". This parameter overrides any other search patterns in a query.

Campaign UI updates

• Users can now specify the branch name during campaign creation. When using the GraphQL API to create a campaign, the branch name is now required.
• Campaign changesets now display labels from code hosts. GitHub labels are currently supported and Bitbucket Server labels are coming soon.

Changes to Sourcegraph’s upgrade policy

Our upgrade policy previously supported upgrades from two minor version increments. We have updated our policy to restrict upgrading to one minor version at a time (e.g., 3.12 -> 3.13). Please reach out to [email protected] if you would like assistance upgrading from a much older version of Sourcegraph. This upgrade policy is now enforced by the sourcegraph-frontend on startup to prevent admins from mistakenly jumping too many versions.

Customization settings

Sourcegraph 3.13 introduces two new customization settings:

• Users who prefer not to see search autocompletion suggestions can hide them by setting search.hideSuggestions to true in their user settings.
• Admins can now set a minimum password length on their instance. Add auth.minPasswordLength to the site config to enforce the password length when users create accounts or update passwords.

Code intelligence and campaigns data added to Sourcegraph pings

Sourcegraph collects a small amount of high-level and aggregate data from each instance. See the complete list of the data we collect, and our ping philosophy and rules. New usage statistics—specifically, aggregate counts of the numbers of hover, go to definition, and find references actions, aggregated 50th, 95th, and 99th percentile latencies for the same actions, and a total count of code change campaigns created, will be sent to Sourcegraph via pings by default. The aggregated event count metrics can be disabled via the site admin flag disableNonCriticalTelemetry.

Experimental features

This release contains several new experimental features. We look forward to hearing your feedback about them! Tweet @sourcegraph or email [email protected] with your thoughts.

Smart search field in plain text mode

The plain text mode search query input field now provides syntax highlighting, hover tooltips, and diagnostics on filters. Set the user or global settings value { "experimentalFeatures": { "smartSearchField": true } } to take advantage of this feature.

Custom git fetch commands

A new experimenal field experimentalFeatures.customGitFetch allows defining custom git fetch commands for code hosts and repositories with special settings.

Import code from any code host with src-expose

src-expose is a new tool that enables admins to import code from any code host. This makes it possible to bring code from other version control systems or textual artifacts from non-version controlled systems (e.g., configuration) into Sourcegraph.

Certificate handling

The new certificates field allows you to add certificates to trust when communicating with a code host (via API or git+http). This is useful for configurations using internal certificate authorities or self-signed certificates. Add { "experimentalFeatures" { "tls.external": { "certificates": ["<CERT>"] } } } to your site config.

Changelog

Added

• Experimental: Added new field experimentalFeatures.customGitFetch that allows defining custom git fetch commands for code hosts and repositories with special settings. #8435
• Experimental: the search query input now provides syntax highlighting, hover tooltips, and diagnostics on filters in search queries. Requires the global settings value { "experimentalFeatures": { "smartSearchField": true } }.
• Added a setting search.hideSuggestions, which when set to true, will hide search suggestions in the search bar. #8059
• Experimental: A tool, src-expose, can be used to import code from any code host.
• Experimental: Added new field certificates as in { "experimentalFeatures" { "tls.external": { "certificates": ["<CERT>"] } } }. This allows you to add certificates to trust when communicating with a code host (via API or git+http). We expect this to be useful for adding internal certificate authorities/self-signed certificates. #71
• Added a setting auth.minPasswordLength, which when set, causes a minimum password length to be enforced when users sign up or change passwords. #7521
• GitHub labels associated with code change campaigns are now displayed. #8115
• When creating a campaign, users can now specify the branch name that will be used on code host. This is also a breaking change for users of the GraphQL API since the branch attribute is now required in CreateCampaignInput when a plan is also specified. #7646
• Added an optional content: parameter for specifying a search pattern. This parameter overrides any other search patterns in a query. Useful for unambiguously specifying what to search for when search strings clash with other query syntax. #6490
• Interactive search mode, which helps users construct queries using UI elements, is now made available to users by default. A dropdown to the left of the search bar allows users to toggle between interactive and plain text modes. The option to use interactive search mode can be disabled by adding { "experimentalFeatures": { "splitSearchModes": false } } in global settings. #8461
• Our upgrade policy is now enforced by the sourcegraph-frontend on startup to prevent admins from mistakenly jumping too many versions. #8157 #7702
• Repositories with bad object packs or bad objects are automatically repaired. We now detect suspect output of git commands to mark a repository for repair. #6676
• Hover tooltips for Scala and Perl files now have syntax highlighting. #8456 #8307

Changed

• experimentalFeatures.splitSearchModes was removed as a site configuration option. It should be set in global/org/user settings.
• Sourcegraph now waits for 90s instead of 5s for Redis to be available before quitting. This duration is configurable with the new SRC_REDIS_WAIT_FOR environment variable.
• Code intelligence usage statistics will be sent back via pings by default. Aggregated event counts can be disabled via the site admin flag disableNonCriticalTelemetry.
• The Sourcegraph Docker image optimized its use of Redis to make start-up significantly faster in certain scenarios (e.g when container restarts were frequent). (#3300, #2904)
• Upgrading Sourcegraph is officially supported for one minor version increment (e.g., 3.12 -> 3.13). Previously, upgrades from 2 minor versions previous were supported. Please reach out to [email protected] if you would like assistance upgrading from a much older version of Sourcegraph.
• The GraphQL mutation previewCampaignPlan has been renamed to createCampaignPlan. This mutation is part of campaigns, which is still in beta and behind a feature flag and thus subject to possible breaking changes while we still work on it.
• The GraphQL field CampaignPlan.changesets has been deprecated and will be removed in 3.15. A new field called CampaignPlan.changesetPlans has been introduced to make the naming more consistent with the Campaign.changesetPlans field. Please use that instead. #7966
• Long lines (>2000 bytes) are no longer highlighted, in order to prevent performance issues in browser rendering. #6489
• No longer requires read:org permissions for GitHub OAuth if allowOrgs is not enabled in the site configuration. #8163

Fixed

• The syntax highlighter (syntect-server) no longer fails when run in environments without IPv6 support. #8463
• After adding/removing a gitserver replica the admin interface will correctly report that repositories that need to move replicas as cloning. #7970
• Show download button for images. #7924
• gitserver backoffs trying to re-clone repositories if they fail to clone. In the case of large monorepos that failed this lead to gitserver constantly cloning them and using many resources. #7804
• It is now possible to escape spaces using \ in the search queries when using regexp. #7604
• Clicking filter chips containing whitespace is now correctly quoted in the web UI. #6498
• Monitoring: Fixed an issue with the Frontend -> Search responses by status panel which caused search response types to not be aggregated as expected. #7627
• Monitoring: Fixed an issue with the Replacer, Repo Updater, and Searcher dashboards would incorrectly report on a metric from the unrelated query-runner service. #7531
• Deterministic ordering of results from indexed search. Previously when refreshing a page with many results some results may come and go.
• Spread out periodic git reclones. Previously we would reclone all git repositories every 45 days. We now add in a jitter of 12 days to spread out the load for larger installations. #8259
• Fixed an issue with missing commit information in graphql search results. #8343

Removed

• All repository fields related to enabled and disabled have been removed from the GraphQL API. These fields have been deprecated since 3.4. #3971
• The deprecated extension API Hover.__backcompatContents was removed.

The changelog for this and previous releases is available on GitHub.

Thank you

Thank you to the many people who contributed to Sourcegraph since the last release!

• @machbio
• @AndreKR
• @philjones
• @mipearson
• @Reagankm
• @marco-c
• @aeb-sia
• @b-j-p
• @CH-JosephBironas
• @Akarshit
• @Donboscow
• @Antoine98

Deploy or upgrade: Local | AWS | DigitalOcean | Kubernetes cluster

From the entire Sourcegraph team (@sourcegraph), happy coding!