Code intelligence team

Sourcegraph Code intelligence team logo

The Code Intelligence team builds tools and services that provide contextual information around code, taking into account its lexical, syntactic, and semantic structure. This includes:

  • An API to provide fast, comprehensive, and accurate answers to important code navigation queries such as Go to Definition and Find References

  • A powerful and flexible language-agnostic model of dependency relationships across projects, repositories, and languages

  • Robust, extensible, and scalable infrastructure to index code across all languages, keep those indexes up-to-date, and efficiently resolve code intelligence queries against all indexed code.

History

The origin of our work is the core of Sourcegraph, first implemented as srclib a code analysis tool developed by our co-founders Beyang and Quinn in 2015.

2016 - We evolved our strategy to utilize LSP (Language Server Protocol) to power the new Sourcegraph code search engine. LSP is a user oriented protocol and requires an implementation of a language server for each programming language.

2017 - Over time we started to experience some issues with language servers. They can be difficult to deploy, slow at runtime, slow to adopt by members of their respective language communities, and slow to develop.

2018 - We added a new basic code intelligence that is built on search-based heuristics to allow us to provide quick and good enough i.e. “imprecise” support for the most popular programming languages.

2019 - While looking for more efficient alternatives to language servers we found LSIF (Language Server Index Format) which provided an index-based implementation of precise code navigation (similar to srclib) with the advantage of being able to build on top of the work done in the LSP community (LSIF and LSP are sister protocols).

2020 and beyond - The Code Intelligence team is responsible for how Sourcegraph understands code and for providing data that powers semantic search, campaigns and code insights. Our API is the point of discovery for the knowledge graph for all source code.

Vision

To be the point of discovery for the knowledge graph for all source code.

Mission

Build a code intelligence platform that understands code in all languages and provides rich metadata to power code search, insights and campaigns.

Goals

See goals and priorities

Contact

Tech stack

Precise code intelligence specific services are written in Go and code intelligence features are added to the existing frontend service where possible. Sourcegraph extensions that provide code intelligence are written in TypeScript. We use SQLite databases to store data specific to a particular LSIF index and Postgres to store all other data.

Team communication

Here are some key ways to contact us:

Team process

The code intel team has defined a set of rituals that keeps the team connected around our common goals. As a team we also believe in good meeting etiquette, and to ensure effective meetings for everyone every meeting must have a meaningful topic, agenda, and stated goal or purpose statement defined ahead of time.

On a weekly basis:

  • Team Sync Meeting: Every Monday, the code intel team meets to stay in sync and hear from stakeholders. We use a Google doc for agenda and meeting notes. Prior to the weekly sync meeting, each team member should:

    1. Add any agenda items that should be discussed.
    2. Review any existing agenda items and be prepared to discuss them.
    3. Update the current release tracking issue with a summary of progress for the previous week and plans for the next week.
  • Team Updates: Every Friday, we update the tracking issue with our weekly progress to inform external stakeholders of the state of milestone goals. The engineers working on a goal are responsible for its update.

For each iteration (currently one month long), we follow this process:

  • Planning Meeting(s): Prior to each iteration the team has planning meetin(s) to discuss which items should be prioritized for the upcoming release. We revisit oustanding work that track against our team goals, and go over our backlog of bugs and feature requests. Once the team agrees on a task, we add it to the tracking issue and as part of our team goals for the milestone.

  • Retrospective Meeting: After the release branch has been cut for the iteration, the team has a retro to discuss how the iteration went, and what changes we might want to make to continously improve our teamwork.

We track most of our work using issues on the Sourcegraph main repository. If you have an issue that wants our attention, mention the @sourcegraph/code-intel team or tag your issue with the team/code-intelligence label.

New Engineer Onboarding

Our team is growing and to help our new teammates have the best onboarding experience we have created additional material that builds on Sourcegraph’s general engineering onboarding guide. We think will help you ramp up in all things code intel.

Weeks 1 - 4

Architecture Presentation: An experienced code intel engineer will give a live presentation primarily to the new hire (and any other Sourcegraph teammate who’s interested) explaining the architecture of our systems at a high level.

Pair Programming: Keeping inline with engineering pairing sessions we will support our new teammates by setting aside dedicated time every week for mentoring. This not only helps deliver guided learning but also is great for team and relationship building.

First Tasks: The team has marked issues we believe are good first issues in Github, following the starter task process work with the code intel engineering manager to select three issues that you think are good ones for you to work on in your first 4 weeks.

Technical Design Docs & Diagrams: We know that a picture is worth a thousand words and we will create visual diagrams to supplement our written documentation for the most critical or complex components and processes. This material is great for async learning and can provide a better understanding of the parts that make up the big picture. Check code intel user documentation and our journey with LSIF post to understand the overall view. Browse through available documentation and diagrams in the code intel developer documentation and code intel Google drive and learn how the Bundle Manager works or view Inter-service request diagrams.

Second Month

Inverted Presentation: Our new teammate has now had some time to explore the source code and will use their knowledge to present to the team. You can choose to either explain the architecture of an existing component or present a solution to a problem you just solved. The team will give feedback and clarify where needed and provide more details to high-level designs.

Feedback: We want to hear from our new teammate how helpful each of these onboarding experiences was and we will continuously update our onboarding guide based on new insights. As per the engineering guide for feedback will be gathered in week 6.

Members

Ownership

On-call

Growth plan

Updated 2020-11-18

We are looking to grow this team a lot to increase our ability to provide precise code intelligence for many languages at massive scale.

If you are interested in scaling our backend systems, please apply to Software Engineer - Backend.

If you are interested to help us expand our language support (for example: building LSIF indexers for languages including, but not limited to, Clang, Java), email [email protected] with how you think you can help.