1. Home
  2. coinbase-eng

coinbase-eng

Phishing attacks are on the rise — here are some steps you can take to protect yourself

Phishing attacks are on the rise — here are some steps you can take to protect yourself

Tl;Dr: As we have discussed in previous blogs, at Coinbase we believe that a healthy and safe crypto industry is critical to growing and maturing the cryptoeconomy. Part of that means sharing information that we believe is useful to protect consumers broadly across the industry. It’s for this reason that we’re sharing the details of a recent large-scale Coinbase-branded phishing campaign that we believe impacted our customers, and many others.

Between April and early May 2021, the Coinbase security team observed a significant uptick in Coinbase-branded phishing messages targeting users of a range of commonly used email service providers (you can learn more about phishing in our Help Center.) Though the attack was broad, it demonstrated a higher degree of success bypassing the spam filters of certain older email services.

The messages used a wide variety of different subject lines, senders, and content. It sometimes sent multiple variations to the same victims. Depending on the variant of email received, different techniques to steal credentials were used as well. The following screenshots show a representative victim experience, but wouldn’t necessarily have been seen in exactly this order by all victims.

This phishing email is designed to alarm the recipient, hoping that they will click the link quickly without taking time to verify other aspects of the message.
For some victims with Hotmail accounts, attackers attempted to add a malicious application to the user’s inbox. If the recipient clicked “Yes”, an attacker would be able to read all the user’s emails (including password reset and device verification emails sent by Coinbase).
Finally, the user would be sent to a Coinbase-branded phishing page attempting to capture the user’s Coinbase login credentials. Note that while the URL contains the word “coinbase”, the domain is not “coinbase.com”

Once the attackers had compromised the user’s email inbox and their Coinbase credentials, in a small number of cases they were able to use that information to impersonate the user to gain access to their Coinbase accounts and bypass our SMS two-factor authentication system. With access to these accounts, the attacker was able to transfer funds to crypto wallets unassociated with Coinbase. We have made changes to prevent this from happening in the future.

Once we learned of the attack, we took a number of steps to protect our customers, including working with external security partners to take down malicious domains and websites associated with the phishing campaign, as well as notifying the email service providers most impacted by the attack.

While attacks like this have the potential to cause significant harm to their victims, taking a few common-sense security precautions can dramatically reduce their efficacy. Below we lay out a few best practices that everyone should follow to keep their various online accounts safe.

Strong passwords

Passwords are the front door locks to your online applications, but far too many people fail when it comes to basic password best practices. In fact, according to a recent study, more than 50% of those polled reused passwords across multiple accounts. If a password is the front door lock to your digital life, reusing a password is like having the same key to your house, car, mailbox and place of work. In other words, reusing a password is barely better than having no password at all. Strong passwords should have a combination of letters (both upper and lower case), numbers and special characters, and be at least 10 characters long. Many password managers will suggest options that meet these standards.

Use a password manager

If you’re following security best practices, you should be using unique, strong passwords for every site you visit, making it impossible to remember your credentials for all of your online accounts. This is where password managers like 1Password and Dashlane become invaluable. Not only will a password manager suggest strong, randomly generated passwords for you, many of them have built-in features that help prevent entering your password on the wrong website — such as those used for phishing campaigns such as the one described above.

Two-Factor Authentication

After choosing a strong, unique password, enabling two-factor authentication (2FA) is the most important way to secure your online accounts. 2FA, also known as 2-step verification, is a security layer in addition to your username and password. With 2FA enabled on your account, you will have to provide your password (first “factor”) and your 2FA code (second “factor”) when signing in to your account. There are many types of 2FA, ranging from a physical key (such as a YubiKey) — the most secure — to SMS verification — the least secure. Many people choose to use SMS 2FA, because it’s linked to a phone number, rather than to one particular device, and is generally the easiest to set up and to use. Unfortunately, that same level of convenience also makes it easier for persistent attackers to intercept your 2FA codes. We strongly encourage everyone that currently uses SMS as a secondary authentication method to upgrade to stronger methods like Google Authenticator or a security key everywhere it is supported.

Question everything

Coinbase, like most financial institutions or FinTech companies, will never contact you asking for your password, two-factor authentication codes, or to take actions like installing new software or sending funds to a cryptocurrency address. Whenever you receive a communication from a company that you have dealings with, check that it is what you think it is. Was it sent from a domain that is consistent with the company? (i.e., help@coinbase.com as opposed to support.coinbase@gmail.com). Is the landing page accurate? (paypal.com as opposed to secure-paypal[.]com). Is the content in the communication or on the website in the same style and of the same quality as previous communications? Many scam sites will have typos, use old logos or have a different feel than their genuine counterparts.

Coinbase provides a number of resources to help our customers avoid online scams and report potential malicious activity. Cryptocurrency transactions are irreversible. If you (or a hacker who has accessed your account illegitimately) send cryptocurrency to a third party, it cannot be reversed or stopped. When you send cryptocurrency to a blockchain address, you must be certain of the legitimacy of any involved third party services and merchants, and only send cryptocurrency to entities you trust. If you’re ever in doubt, do some research and never send funds or share information unless you are 100% certain that you are interacting with who you think you are.


Phishing attacks are on the rise — here are some steps you can take to protect yourself was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Latam Insights: Paraguay Shuts Down Largest Illegal Bitcoin Mining Farm to Date, Argentina and El Salvador Discuss Bitcoin

Web3 Will Be Driven By Crypto’s Networked Collaboration

And What We Can Learn From The Linux Development Model

By Harry Alford, Business Development Manager, Coinbase Cloud

Where Web 1.0 was the Static Web, and Web 2.0 is the Social Web, Web 3.0 will be the Decentralized Web. It will move us from a world where communities contribute but don’t own or profit to one where they can through collaboration. By breaking away from traditional business models centered around benefiting the corporation, Web3 foretells the possibility of community-centered economies of scale. This collaborative spirit, and its associated incentive mechanisms, attracts some of the most talented and ambitious developers working in technology today, unlocking projects never previously possible.

Web3, as Ki Chong Tran once said, is “the next major iteration of the Internet, which promises to wrest control from the centralized corporations that today dominate the web.” Web3-enabled collaboration is made possible by decentralized networks that no single entity controls. What might we expect in these nascent developmental stages of a technology that is collaborative instead of competitive?

In closed-source business models, users trust a business’s people to manage funds and execute services. With open-source projects, users trust the technology to perform these tasks. In Web2, the bigger network wins. In Web3, whoever builds the biggest network together wins.

In a decentralized world, not only is participation open to all, the incentive structure is designed so that the more people who participate, the more everybody succeeds.

Learning from Linux

Linux, the open-source software created in 1991 that is behind a majority of Web2’s websites, was paradigm-shifting for how the Internet (Web2) was developed and provides a clear example of how collaborative processes have the power to drive the future of all technology development. Linux wasn’t developed by an incumbent tech giant, but by a group of volunteer programmers utilizing networked collaboration. Networked collaboration is when autonomous people freely share information without central control, supported instead by code or blockchains.

In The Cathedral And The Bazaar, author Eric S. Raymond shares his observations of the Linux kernel development process and his experiences managing open source projects. During the rise of the Internet, Raymond depicts a time when the popular mindset was to develop complex operating systems carefully coordinated by a small, exclusionary group of people — “cathedrals.” Cathedrals are the traditional corporations and financial institutions, with long releases that take even longer to perfect.

Linux evolved in a completely different way according to Raymond, explaining, “quality was maintained not by rigid standards or autocracy but by the naively simple strategy of releasing every week and getting feedback from hundreds of users within days, creating a sort of Darwinian selection on the mutations introduced by developers. To the amazement of almost everyone, this worked quite well.” This Linux development model, or “bazaar” model as Raymond terms it, assumes that “bugs are generally shallow phenomena” when exposed to an army of hackers without significant coordination.

The open development policy of Linus Torvalds, who conceived the Linux kernel, was far from cathedral-building. Linus’s users were his co-developers. Cultivating a community while leveraging collaboration scales innovation up to a degree of intensity matching the users’ complexity. Linus did a lot of things right, like releasing early and often, growing a beta list of users, sending announcements encouraging participation, and he listened to (and gave credit to) beta testers.

As Linux represents software built by networked participants rather than a single cathedral, Web3 represents an Internet run by vast swathes of networked participants instead of Web2’s version dominated by a controlling few.

Proof of stake: the collaborative testing ground of Web3

Crypto is a technology model for building projects where the user is centered rather than the companies. The incentives for making something secure and usable are made possible by everyone. One area where every user maintains some level of participation is on proof-of-stake protocols — which are also the fastest growing industry sector. Upgrades to the Ethereum network could turn crypto staking into a $40 billion industry by 2025, according to JPMorgan. With proof of stake, there are levels of participation that prevent the “cathedral” from happening.

Even though there’s a team behind every chain, ultimately, the design of a decentralized network means that they will not have absolute control. At different levels of participation, multiple groups of users play critical roles in harmony with key features like governance and scalability.

As developers transition away from proof of work for greater power efficiency and scalability more protocols will implement a version of proof of stake, in which users stake tokens (cryptocurrency) to run validators for a particular network. Whether service providers or retail users, commercial or institutional, users are incentivized to secure the network by locking up tokens and are rewarded in the form of the network’s native tokens. Validators can be slashed for misbehavior like double signing blocks or downtime. Penalties include loss of existing funds and missing out on future rewards. Staking is your voice, good behavior is encouraged to increase the number of participating users, and the network can flourish securely without the need for centralized control.

Since proof of stake protocols are in relatively early-stage development and are newer than proof of work protocols like Bitcoin, frequent upgrades take place. Operators and co-developers are relied upon for network updates and proposals. For instance, Polkadot, a Web3 platform, is a sharded protocol that enables blockchain networks to operate together seamlessly. Polkadot utilizes user-driven governance to upgrade the network. The loose coordination is “on-chain and enacted autonomously, ensuring that Polkadot’s development reflects the values of the community and avoids stagnation.” When the network sees limitations of current staking parameters, they impose restrictions to ensure the stability of the network. The community changes the terms of service, not the company.

Rewards are the incentive mechanism for people who run nodes, but not every chain draws users for the primary purpose of making a financial return. There needs to be memetics, culture, socialization, and purpose for people to use the network. Blockchains will continue to be abstracted away in Web3. The millions of customers buying NFTs from OpenSea, the first NFT marketplace to pass $1 billion in monthly trading volume, probably don’t care that it supports Polygon, “a popular Layer 2 Ethereum blockchain that boasts a more energy-efficient structure that will allow OpenSea to entirely eliminate gas fees for creators, buyers, and sellers on that blockchain,” according to Techcrunch. Nor do they ultimately care that Degenerate Ape Academy is built on Solana, a proof-of-stake blockchain. The collectors only care about adding rare pictures of cartoon apes to their portfolios. They participate in what offers the memetic value to them that is created by other users on top of the underlying protocol. Unbeknownst to most crypto gamers and art collectors, the more usage there is, the stronger the network becomes.

While abstraction might temporarily bring new users into the space, institutional users want to be reassured for the long-term by solid technology, which means, in some cases, the blockchain supports cathedrals. Provenance is a proof-of-stake blockchain network developed by Figure, a home equity line of credit originator, to help financial service industry needs by providing a ledger, registry, and exchange across multiple financial assets and markets. The chain is run and operated by large financial institutions like Franklin Templeton and Caliber Home Loans.

Financial institutions, despite their traditional practices of centralizing control, are incentivized to run nodes because they receive transaction fees. They’re also incentivized to participate because certain aspects of their business are cheaper using Web3 and blockchain technology than through their old Web2 processes. Even as a cathedral, they have unique incentives powered by a proof of stake protocol.

As decentralized teams play less of a role in managing the protocols, the individual participants manage the protocol requiring only minor organization. However, participants will still need to be incentivized to participate in securing, using, and building on the protocol. All of these examples amplify what makes the bazaar and Web3’s networked collaboration so successful — an alliance of autonomous users receiving constant stimulation while working towards a common goal.

In this case, the medium is the blockchain: network that stores, transfers, and exchanges value. As Raymond stated, “Linus was keeping his hackers/users constantly stimulated and rewarded — stimulated by the prospect of having an ego-satisfying piece of the action. Rewarded by the sight of constant (even daily) improvement in their work.”

Linux was a catalyst for the creative thinking that’s manifesting itself in Web3, today and in its exciting future. Cathedrals, however, still exist and it’s unclear if Web2 will fully transition to Web3. How can technology truly belong to the people and Web3 be wholly driven by networked collaboration? Does it have to be zero-sum? It’s a leap of faith. Cathedrals that enable rent-seeking behavior, and are so ingrained in Web2 economic and business models, will need to continue to embrace the bazaar, bending towards how these communities work from the bottom-up, not top-down.

Web3 might not be the final answer, but it’s the current iteration, and innovation isn’t always obvious in the beginning. Loot “started with lists of fictional gear, aided by ample decentralized imagination.” As long as talented builders are incentivized to collaborate, Web3 can open up a whole new world we’ve never dreamed possible.

The bazaar model partially remade the commercial software world in its image like crypto’s networked collaboration will inevitably do to traditional business models. Centralized organizations will not win without embracing decentralization to some degree, and those that do will be decades ahead of their competition while possibly doing some good.

Networked collaboration harnesses the brainpower of communities and has plausible promise. We’ve seen an explosion of interest in crypto that continues to parallel, if not follow, the principles laid out by Linus Torvalds. The future of the Internet will belong to the “people who leave behind the cathedral and embrace the bazaar.”


Web3 Will Be Driven By Crypto’s Networked Collaboration was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Latam Insights: Paraguay Shuts Down Largest Illegal Bitcoin Mining Farm to Date, Argentina and El Salvador Discuss Bitcoin

How Coinbase interviews for engineering roles

Coinbase is on a mission to increase economic freedom in the world. In order to achieve this mission, we need to build a high performing team. Our top priority is attracting and retaining great talent. and we take extraordinary measures to have exceptional people in every seat. This post is designed to give candidates a sneak preview of what we look for in our technical interview process.

In this post we’ll focus on what we look out for in an engineering candidate. Some of this advice will apply to other positions too, but it’s most useful if you want to join us as an engineer.

When joining the interview process you’ll progress through a series of stages. In each stage we’ll assess you in different ways to ensure the role you’re interviewing for is a good mutual fit. While the exercises and questions you face will vary, we always look out for the Coinbase cultural tenets:

  • Clear communication
  • Efficient execution
  • Act like an owner
  • Top talent
  • Championship team
  • Customer focus
  • Repeatable innovation
  • Positive energy
  • Continuous learning

Learn more about these tenets here. You may not get an opportunity to display all of these qualities at every interview stage, but this will give you a good idea of what we are looking for. When we assess your performance we will do so almost exclusively through the lens of these tenets.

The interview stages are (typically but not always):

  • an initial chat with someone from HR about the role
  • one 60 minute pair programming problem
  • one or two 60 minute engineering manager interviews
  • one or two 60 or 90 minute pair programming interviews
  • one or two 60 minute system design interviews

You will need to perform well at all stages to get an offer, so it’s important to prepare for each interview stage. That said, most people that are unsuccessful in their Coinbase interview loop fail on the pair programming stages. Let’s start there.

Pair Programming

In the pair programming interview(s) you will work through a problem with one of our engineers. To start, your interviewer will provide you with a short brief of the problem. Now it’s up to you to code a solution to the problem.

It’s not enough to solve the problem to pass this stage. We are not looking for a minimal Leetcode-style optimal solution. We are looking for evidence that you are able to produce production-grade code. As a result, we assess both the end result and how you got to the result, giving credit for both components. If you get stuck on a bug, how do you overcome it? Do you know your tooling well? Do you use the debugger with a breakpoint, or do you change random lines of code until it works? Is there a method to how you approach a coding problem?

We will look beyond the completeness and correctness of your solution. We will assess the quality and maintainability of your code, too. Is your code idiomatic for your chosen language? Is it easy to read through and understand? What about variable naming? Do you leverage the tooling that is available to you in your IDE and terminal? How can we be confident that your code is correct? Did you test it?

How well do you understand the problem? Do you ask relevant clarifying questions? How well do you take the interviewer’s feedback?

Don’t be discouraged if you do not reach the very end of the problem. We design our interview problems to fill more than the allotted time. The interviewer will stop the interview after either 90 minutes have passed, or when they are confident in their assessment. Ending an interview early is not always a bad sign.

Most candidates who fail the interview do so because their code or process isn’t good enough. We do not typically fail you for an incomplete solution.

Let’s look at a practical example. Suppose the problem is:

Given a list that contains all integers from 1 to n — 1. The list is not sorted. One integer occurs twice in this list. Write a function that determines which.

Here’s the first example solution (there are errors!):

def duplicates(integers):
"""duplicates takes a list of integers and returns the first duplicate value or None if all values are unique"""
 if not isinstance(integers, list):
  raise ArgumentError(“expected list as input”)
sorted_integers = integers.sort()
previous_value = nil
for x in sorted_integers:
 if x == previous_value:
  return x
 previous_value = x
 return None
def test_duplicates():
 assert duplicates([]) == None, "empty array is considered unique"
 assert duplicates([1, 2]) == None , "array of unique values returns None"
 assert duplicates([1, 2, 2]) == 2, "duplicate returns the duplicate integer"
 assert duplicates([1, 2, 2, 1]) == 2, "multiple duplicates returns the first duplicate"

And the second solution (there are errors here, too!):

def dupilcateIntegers(l ):
 n = len(l)
 return sum(l) - ((len(l)+1) * len(l))/2

The first solution doesn’t actually solve the problem. But the author seems to have thought about error handling and input validation. The candidate has thought about the problem and its edge cases. Furthermore the solution attempts to also solve for a larger and more general class of the same problem. They’ve added a short docstring and the code is generally well-formatted and idiomatic python. We would be inclined to consider the first solution a pass. There’s a bug in the implementation and the algorithm is not optimal yet the code is maintainable and generally well structured (with tests too!). This solution is good.

The second solution is correct and optimal, yet this candidate would not pass the interview. The formatting is sloppy and there are spelling mistakes, and unused variables. The code itself is terse and difficult to understand. This candidate would probably be rejected.

Finally, also keep in mind that you have only 90 minutes to complete the problem. Our problems don’t really have any tricks in them, and the naive solution is typically good enough. We won’t ask you to invert a binary tree, but we will ask you to solve a simplified version of a real life problem. We’re looking for production-like code, not hacky solutions.

So how would you best prepare for the pair programming interview with us? Don’t focus too much on grinding Leetcode. It’s better to focus on the fundamentals. Learn your editor, your debugger, and your language. Practice writing well formatted and well structured code with relevant method and variable names, good state management and error handling.

System Design

In our system design interview you will be asked to design the general architecture of a real-world service. For example: How would you design a Twitter feed?

The brief is typically short, and it’s up to you to ask the interviewer for clarifications around the requirements.

Don’t dive too deeply into any one specific aspect of the design (unless asked by the interviewer). It’s better to keep it general and give a specific example of a technology you know well, that would be a good fit for the use case at hand. Example: “For this service an RDBMs database would be a good choice, because we don’t know exactly what the queries will look like in advance. I would choose MariaDB.”

Be sure to address the entire problem, and if you’re unsure if you’ve covered everything ask the interviewer to clarify, or if there’s anything they’d like you to expand upon.

If you are unsure about the specifics of a particular component in your design, it’s best to try to let your interviewer know and to tell them how you would go about finding the answer. Don’t wing it — being wrong with confidence is a negative signal, whereas humility is a positive signal. A good answer might be: “I don’t know if the query pattern and other requirements fit well with an SQL database here, but I have the most experience with MariaDB so it would be my default choice. However, before making a decision I would have to research what its performance might look like in this specific case. I’d also research some NoSQL alternatives like MongoDB and perhaps also a column wide store like Cassandra.”

You’ll be assessed on your ability to explore the requirements, and how well your design might perform in real life. Do you take scalability into account? How about error handling and recovery? Do you design for change? Have you thought about observability? You’ll also be assessed on how well you communicate your design and thoughts to the interviewer.

General Tips

During our interview process, we look for signals that help us understand whether there is a skill match but more importantly a cultural fit. Some of the signals we look for:

  1. Be honest — Honesty always pays. If you’ve seen the question before, best to let your interviewer know so that an alternate question can be discussed. Similarly, exaggerating current scope/responsibilities is considered a red flag.
  2. Speak your mind — Even if the question might seem difficult or you need time to think, vocalize your thoughts so that the interviewer can help you along. It’s not as important to get the right answer as it is to have a reasonable thought process.
  3. Understand before responding — It’s best to listen and understand the question before responding. If you’re unsure, ask to clarify or state your assumptions. We aren’t looking for a quick response but always appreciate a thoughtful response. If the question isn’t clear the first time, feel free to request the interviewer to repeat it.
  4. Good Setup — Being a remote first company, our interviews are virtual on Google Meet. Take the meeting at a place where the internet connection and your audio/video is good. It’s better to reschedule in advance if your setup isn’t tip-top. Finally, test your microphone and camera an hour before joining the call. We keep our cameras on the entire interview and expect you to do the same.
  5. Be Prepared — We advise that you go through the links your recruiter shares with you as part of the interview invite. They contain information about how everyone at Coinbase operates and what they value.
  6. Ask what’s on your mind — Our panelists always leave time for the candidates to ask questions.Take that opportunity to ask questions that would help you decide your Coinbase journey rather than asking generic questions (most of which are answered on our blog). You will interview with engineers and managers so tailor your questions to the unique perspectives offered by each role.
  7. Crypto or Industry knowledge — Unless you are specifically interviewing for a role that requires deep crypto/blockchain knowledge (your recruiter would be able to share this with you), we aren’t looking for this knowledge as a mandatory skill. As long as you are willing to learn, we want to talk to you — even if you are entirely new to crypto. We all were new at one point too!

Thanks for taking the time to learn a little bit more about our interview process. If you are interested in building the future of finance, have a look at our open roles here. Good luck!


How Coinbase interviews for engineering roles was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Latam Insights: Paraguay Shuts Down Largest Illegal Bitcoin Mining Farm to Date, Argentina and El Salvador Discuss Bitcoin

Remote-First Engineering: What We’ve Learned One Year In

By Manish Gupta, EVP, Engineering

When we announced last May that Coinbase would embrace being a remote-first company, we knew there was much we hadn’t yet figured out. How would we tap into a capable, international engineering community without losing the sense of culture and mission that drives us at Coinbase? There wasn’t a playbook for us to consult, and the challenges of growing during a pandemic were many.

Since that time, we’ve hired, onboarded, and integrated hundreds of new engineers, more than doubling our team year over year: we’re now well on the path to becoming a decentralized, globally distributed engineering organization. But ensuring that we could scale, and not simply grow, has been a constant challenge. We’d like to share a few of the things we’ve learned along the way.

  1. Great Talent Is Everywhere

The tech industry sometimes seems to forget this, but no one city or country has a monopoly on great talent. By looking across the US and internationally, we’ve been able to find extremely strong engineering teams that had no interest in moving to downtown San Francisco — no matter how interesting the opportunity. Since we started hiring anywhere, only 36% of our hires are in SF or NYC, and nearly one in six new hires is outside of the United States. As a result, we can assemble global teams with representation across diverse geographies and backgrounds.

2. Commit To A Regional Approach

Instead of defining teams by specific countries or cities, we’ve organized our engineering organization into a smaller set of much larger geographies that we call “Regions”. Each Region is laid out longitudinally and maps to a set of adjacent time zones. We plan to have only three such Regions at this time, mapping to the Americas, EMEA and APAC. As a result, teams get the flexibility to work on a broad range of projects, with interesting work distributed globally.

To maximize the common working hours for a given team, we try to keep managers and their direct reports within the same Region, even if team members are spread out across the region. Because the regions share working times but not artificial borders, they’re neither so small that they can’t draw on a rich talent bench, nor so big that they can’t collaborate effectively.

3. Autonomous Teams Are Effective Teams

Some companies try to outsource small slices of their existing work, like shared services or backend infrastructure, to their remote teams. It quickly became apparent that this wouldn’t work if we wanted to attract high-performing engineers. High performers want to be challenged, and they seek end-to-end ownership. Our teams at Coinbase have the flexibility of working on many types of projects, and over time they’ll cover a diverse range of domains. We’re also co-locating our product management, engineering, and design teams in-region to ensure that they can collaborate effectively in building cohesive product experiences. Whether supporting our global platform or internationalizing our products, we want to make sure regional teams can make progress in these areas independently.

4. One Size Fits None

Although we’re excited at the opportunities that being remote-first brings, we’ve also learned how important it is to tailor our talent approach to local needs. For example, we’ve (re)discovered the importance of working with local experts and partners to better understand what our employees and customers need; we found some of the conventional wisdom we first heard to be misleading. We’ve also built specially customized onboarding programs to help new hires acclimate, like our “Cikka” program in India. Cikka, which stands for “Coinbase Sikka” (“Coin” in many Indian languages) provides a US$1,000 crypto stipend to new employees that they can use to explore Coinbase’s range of products.

5. Culture-Building Is Everyone’s Responsibility

Finally, much ink has been spilled about the importance of culture-building in a remote environment. But for us, this isn’t a temporary pandemic adjustment, but a sea change in how we interact with our colleagues. As a result, we’re investing significant time and resources into continuously improving how we create a global Coinbase culture. For example, we rebuilt our engineering onboarding program, “Bootcamp,” with an updated curriculum, virtual social activities, and “lunch and learns” with global leadership to make sure people can integrate more seamlessly into our borderless organization. We’ve invested in scaling our mentorship program to provide new engineers with opportunities for 1:1 support. We’re experimenting with new communication mechanisms like video newsletters. And we’ve worked hard to ensure a global internship program can succeed with the right mentorship and infrastructure in place. (It seems to be working: of our class of 84 interns in 2020, 67 returned full-time this year!)

Naturally, we still have a long way to go and much to learn. But we’re confident that this approach is creating a stronger engineering organization and culture at Coinbase, and that we’ll continue to scale as we help to increase economic freedom in the world. If you’re interested in joining to help us do that, check out our open roles!


Remote-First Engineering: What We’ve Learned One Year In was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Latam Insights: Paraguay Shuts Down Largest Illegal Bitcoin Mining Farm to Date, Argentina and El Salvador Discuss Bitcoin

How we scaled data streaming at Coinbase using AWS MSK

By: Dan Moore, Eric Sun, LV Lu, Xinyu Liu

Tl;dr: Coinbase is leveraging AWS’ Managed Streaming for Kafka (MSK) for ultra low latency, seamless service-to-service communication, data ETLs, and database Change Data Capture (CDC). Engineers from our Data Platform team will further present this work at AWS’ November 2021 Re:Invent conference.

Abstract

At Coinbase, we ingest billions of events daily from user, application, and crypto sources across our products. Clickstream data is collected via web and mobile clients and ingested into Kafka using a home-grown Ruby and Golang SDK. In addition, Change Data Capture (CDC) streams from a variety of databases are powered via Kafka Connect. One major consumer of these Kafka messages is our data ETL pipeline, which transmits data to our data warehouse (Snowflake) for further analysis by our Data Science and Data Analyst teams. Moreover, internal services across the company (like our Prime Brokerage and real time Inventory Drift products) rely on our Kafka cluster for running mission-critical, low-latency (sub 10 msec) applications.

With AWS-managed Kafka (MSK), our team has mitigated the day-to-day Kafka operational overhead of broker maintenance and recovery, allowing us to concentrate our engineering time on core business demands. We have found scaling up/out Kafka clusters and upgrading brokers to the latest Kafka version simple and safe with MSK. This post outlines our core architecture and the complete tooling ecosystem we’ve developed around MSK.

Configuration and Benefits of MSK

Config:

  • TLS authenticated cluster
  • 30 broker nodes across multiple AZs to protect against full AZ outage
  • Multi-cluster support
  • ~17TB storage/broker
  • 99.9% monthly uptime SLA from AWS

Benefits:

Since MSK is AWS managed, one of the biggest benefits is that we’re able to avoid having internal engineers actively maintain ZooKeeper / broker nodes. This has saved us 100+ hours of engineering work as AWS handles all broker security patch updates, node recovery, and Kafka version upgrades in a seamless manner. All broker updates are done in a rolling fashion (one broker node is updated at a time), so no user read/write operations are impacted.

Moreover, MSK offers flexible networking configurations. Our cluster has tight security group ingress rules around which services can communicate directly with ZooKeeper or MSK broker node ports. Integration with Terraform allows for seamless broker addition, disk space increases, configuration updates to our cluster without any downtime.

Finally, AWS has offered excellent MSK Enterprise support, meeting with us on several occasions to answer thorny networking and cluster auth questions.

Performance:

We reduced our end-to-end (e2e) latency (time taken to produce, store, and consume an event) by ~95% when switching from Kinesis (~200 msec e2e latency) to Kafka (<10msec e2e latency). Our Kafka stack’s p50 e2e latency for payloads up to 100KB averages <10 msec (in-line with LinkedIn as a benchmark, the company originally behind Kafka). This opens doors for ultra low latency applications like our Prime Brokerage service. Full latency breakdown from stress tests on our prod cluster, by payload size, presented below:

Proprietary Kafka Security Service (KSS)

What is it?

Our Kafka Security Service (KSS) houses all topic Access Control Lists (ACLs). On deploy, it automatically syncs all topic read/write ACL changes with MSK’s ZooKeeper nodes; effectively, this is how we’re able to control read/write access to individual Kafka topics at the service level.

KSS also signs Certificate Signing Requests (CSRs) using the AWS ACM API. To do this, we leverage our internal Service-to-Service authentication (S2S) framework, which gives us a trustworthy service_id from the client; We then use that service_id and add it as the Distinguished Name in the signed certificate we return to the user.

With a signed certificate, having the Distinguished Name matching one’s service_id, MSK can easily detect via TLS auth whether a given service should be allowed to read/write from a particular topic. If the service is not allowed (according to our acl.yml file and ACLs set in ZooKeeper) to perform a given action, an error will occur on the client side and no Kafka read/write operations will occur.

Also Required

Parallel to KSS, we built a custom Kafka sidecar Docker container that: 1) Plugs simply into one’s existing docker-compose file 2) Auto-generates CSRs on bootup and calls KSS to get signed certs, and 3) Stores credentials in a Docker shared volume on user’s service, which can be used when instantiating a Kafka producer / consumer client so TLS auth can occur.

Rich Data Stream Tooling

We’ve extended our core Kafka cluster with the following powerful tools:

Kafka Connect

This is a distributed cluster of EC2 nodes (AWS autoscaling group) that performs Change Data Capture (CDC) on a variety of database systems. Currently, we’re leveraging the MongoDB, Snowflake, S3, and Postgres source/sink connectors. Many other connectors are available open-source through Confluent here

Kafdrop

We’re leveraging the open-source Kafdrop product for first-class topic/partition offset monitoring and inspecting user consumer lags: source code here

Cruise Control

This is another open-source project, which provides automatic partition rebalancing to keep our cluster load / disk space even across all broker nodes: source code here

Confluent Schema Registry

We use Confluent’s open-source Schema Registry to store versioned proto definitions (widely used along Coinbase gRPC): source code here

Internal Kafka SDK

Critical to our streaming stack is a custom Golang Kafka SDK developed internally, based on the segmentio/kafka release. The internal SDK is integrated with our Schema Registry so that proto definitions are automatically registered / updated on producer writes. Moreover, the SDK gives users the following benefits out of the box:

  • Consumer can automatically deserialize based on magic byte and matching SR record
  • Message provenance headers (such as service_id, event_time, event_type) which help conduct end-to-end audits of event stream completeness and latency metrics
  • These headers also accelerate message filtering and routing by avoiding the penalty of deserializing the entire payload

Streaming SDK

Beyond Kafka, we may still need to make use of other streaming solutions, including Kinesis, SNS, and SQS. We introduced a unified Streaming-SDK to address the following requirements:

  • Delivering a single event to multiple destinations, often described as ‘fanout’ or ‘mirroring’. For instance, sending the same message simultaneously to a Kafka topic and an SQS queue
  • Receiving messages from one Kafka topic, emitting new messages to another topic or even a Kinesis stream as the result of data processing
  • Supporting dynamic message routing, for example, messages can failover across multiple Kafka clusters or AWS regions
  • Offering optimized configurations for each streaming platform to minimize human mistakes, maximize throughput and performance, and alert users of misconfigurations

Upcoming

On the horizon is integration with our Delta Lake which will fuel more performant, timely data ETLs for our data analyst and data science teams. Beyond that, we have the capacity to 3x the number of broker nodes in our prod cluster (30 -> 90 nodes) as internal demand increases — that is a soft limit which can be increased via an AWS support ticket.

Takeaways

Overall, we’ve been quite pleased with AWS MSK. The automatic broker recovery during security patches, maintenance, and Kafka version upgrades along with the advanced broker / topic level monitoring metrics around disk space usage / broker CPU, have saved us hundreds of hours provisioning and maintaining broker and ZooKeeper nodes on our own. Integration with Terraform has made initial cluster configuration, deployment, and configuration updates relatively painless (use 3AZs for your cluster to make it more resilient and prevent impact from a full-AZ outage).

Performance has exceeded expectations, with sub 10msec latencies opening doors for ultra high-speed applications. Uptime of the cluster has been sound, surpassing the 99.9% SLA given by AWS. Moreover, when any security patches take place, it’s always done in a rolling broker fashion, so no read/write operations are impacted (set default topic replication factor to 3, so that min in-sync replicas is 2 even with node failure).

We’ve found building on top of MSK highly extensible having integrated Kafka Connect, Confluent Schema Registry, Kafdrop, Cruise Control, and more without issue. Ultimately, MSK has been beneficial for both our engineers maintaining the system (less overhead maintaining nodes) and unlocking our internal users and services with the power of ultra-low latency data streaming.

If you’re excited about designing and building highly-scalable data platform systems or working with cutting-edge blockchain data sets (data science, data analytics, ML), come join us on our mission building the world’s open financial system: careers page.


How we scaled data streaming at Coinbase using AWS MSK was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Latam Insights: Paraguay Shuts Down Largest Illegal Bitcoin Mining Farm to Date, Argentina and El Salvador Discuss Bitcoin

Top ten smart contract security risks

By the Blockchain Security Team at Coinbase

Securing smart contracts from risks remains hard. Unaddressed security vulnerabilities readily turn into existential threats to your token’s viability. So how can asset issuers prevent smart contract vulnerabilities from leading to real financial losses on token networks?

Keep users’ tokens and token networks safe from attackers by teaching developers to write smart contracts and design robust testing based on this list of ERC-20 implementation risks.

In Introducing Solidify, we shared how the Coinbase blockchain security team performs smart contract vulnerability review at scale. A meta analysis across a few hundred token Solidify security reports resulted in a list of most frequent and severe risks based on potential impact to token network security.

The top ten Smart Contract Risks (SCR) fall into three categories:

  1. Operational Risks — Authorization features that are exploited when token network governance is insufficient or flawed
  2. Implementation Risks — Intrinsic errors that result in unintended smart contract behavior
  3. Design Risks — Accepted system features that are exploited to alter intended smart contract behavior

OPERATIONAL RISKS

SCR-1: Super User Account or Privilege Management

The smart contract implements functions that allow a privileged role to unilaterally and arbitrarily alter the functionality of the asset.

SCR-2: Blacklisting and Burning Functions

The smart contract implements functions that allow a privileged role to prohibit a specific address from exercising an essential functionality.

SCR-3: Contract Logic or Asset Configuration can be arbitrarily changed

The smart contract implements functions that allow the holder of a privileged role to unilaterally and arbitrarily alter the functionality of the asset.

SCR-4: Self-Destruct Functions

The smart contract implements a function that allows a privileged role to remove the token contract from the blockchain and destroy all tokens created by the contract.

SCR-5: Minting Functions

The smart contract implements a function that allows a privileged role to increase a token’s circulating supply and/or the balance of an arbitrary account.

IMPLEMENTATION RISKS

SCR-6: Rolling Your Own Crypto and Unique Contract Logic

The smart contract implements functions that allow the holder of a privileged role to unilaterally and arbitrarily alter the functionality of the asset.

SCR-7: Unauthorized Transfers

The smart contract contains functions that circumvent standard authorization patterns for sending tokens from an account.

SCR-8: Incorrect Signature Implementation or Arithmetic

The smart contract contains operations that can result in unexpected contract states or account balances.

DESIGN RISKS

SCR-9: Untrusted Control Flow

The smart contract invokes functions on different smart contracts in order to trigger functionality not defined within the contract itself.

SCR-10: Transaction Order Dependence

The smart contract allows asynchronous transaction processing that can be exploited for profit or protocol correctness through mempool transaction reordering.

For Coinbase customer funds’ safety, the Coinbase blockchain security team assesses all tokens being considered for listing for proper risk mitigations according to the above vulnerabilities. If you’re looking to get a token listed on Coinbase, we encourage you to check your token’s security by reviewing and testing for the aforementioned risks.

Future posts will help you review your token’s security by examining the top Smart Contract Risks in detail and will also provide countermeasure recommendations.

If you are interested in listing your token with Coinbase, visit the Coinbase Asset Hub. If you are interested in securing the future of finance, Coinbase is hiring.


Top ten smart contract security risks was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Latam Insights: Paraguay Shuts Down Largest Illegal Bitcoin Mining Farm to Date, Argentina and El Salvador Discuss Bitcoin

Introducing Solidify — a tool to automatically detect and classify smart contract security risks

Introducing Solidify — a tool to automatically detect and classify smart contract security risks

By Peter Kacherginsky, Principal Blockchain Security Engineer

When our blockchain security team started doing a few smart contract security reviews in 2018, we had no idea that we would be doing hundreds of reviews in 2021.

To grow the open financial system, Coinbase is committed to expanding its list of supported cryptocurrencies. Recently, we launched the Asset Hub to streamline our diligence process for new listings. One of the critical factors we consider is smart contract security, especially with regard to Ethereum-based tokens which can vary greatly in technical complexity. Therefore, we created a smart contract security tool called Solidify to help automate, standardize, and scale this process.

Manual smart contract analysis is a time intensive and error prone process. Experienced teams miss occasional vulnerabilities which can lead to significant monetary loss. To keep our customers and Coinbase safe, our token listing process requires security reviews and risk mitigation recommendations for every smart contract. Consider our challenge of figuring out how to do this specialty risk identification and recommendation process at scale. We identified existing smart contract security scanners that are geared toward manual reviews. But these tools do not offer the required safety guarantees and standardized risk scoring for fully automated analysis.

To solve this problem we developed a tool called Solidify (a play on Solidity) to increase the rate of new asset security reviews without lowering our high security standard that Coinbase customers have come to expect for protecting their tokens. Solidify uses a large signature database and a pattern matching engine to reliably detect contract features and their risks, standardize and score smart contract risks, suggest mitigation strategies, and generate detailed reports to help inform our decision on whether or not Coinbase should list the asset. Solidify evaluates security risks of hundreds of smart contracts either fully automatically or through identification of unique functions that require additional manual review.

How Solidify Works

Most smart contract risks come from operations design choices by asset issuers that introduce potentially dangerous functionality (e.g. freezing, upgrading, etc.) or non-standard function implementations that are insufficiently tested (e.g. custom withdrawal logic). On the design side, we observe most asset issuers using standard patterns such as OpenZeppelin’s contracts library to implement features like asset pausing:

contract Pausable is Ownable {
event Pause();
event Unpause();
  bool public paused = false;
  modifier whenNotPaused() {
require(!paused);
_;
}
  modifier whenPaused() {
require(paused);
_;
}
   function pause() onlyOwner whenNotPaused public {
paused = true;
Pause();
}
   function unpause() onlyOwner whenPaused public {
paused = false;
Unpause();
}
}

Source: openzeppelin-contracts-1.3.0/contracts/lifecycle/Pausable.sol

By using the above pause() functions, the owner of the smart contract can halt all send/receive operations and, in some cases, negatively impact the contract utility. By the presence of the pause library and respective transfer functions which use them, we can reliably classify the contract having a risk of getting paused. Solidify does this by aggregating all known instances and possible variations on the pause() function and automatically checking if the currently analyzed contract has it. Unique signatures are generated using AST (Abstract Syntax Tree) structures to avoid duplicates produced by slight variations in syntax while ensuring grammatical accuracy of what the function actually does.

The above sample AST tree for the pause() function will be processed to generate a unique hash which will be used as a signature to match against other smart contracts:

Below is a sample entry in the signature database generated by Solidify for the pause() function:

{
"Hash": 
"05654d81921af71079698aff1b4be500d1405ece9364a05915b738376c0250b5",
"Name": "pause()",
"Mutability": "nonpayable",
"Source": ["signatures/openzeppelin/openzeppelin-contracts-1.10.0/contracts/lifecycle/Pausable.sol:666:90:0",  
"signatures/openzeppelin/openzeppelin-contracts-1.11.0/contracts/lifecycle/Pausable.sol:666:90:0",  
"signatures/openzeppelin/openzeppelin-contracts-1.9.0/contracts/lifecycle/Pausable.sol:666:90:0"
[.. redacted ..]
],
 "Doc": "@dev called by the owner to pause, triggers stopped stater",
"Features": [
"pausing"
 ]
},

Notice that the same pause() function signature is present across 3 different releases of the OpenZeppelin library and other tokens that included that code verbatim. The above signature also includes a list of features such as “pausing” which allows us to automatically evaluate asset’s risk and suggest mitigations (e.g. robust multi-sig for the owner account). Solidify performs similar signature matching against every single function in an analyzed contract until all functions are either detected or added to our signature database so we never need to analyze the same code twice.

Solidify currently has about 6,000 unique signatures which are used to efficiently match risks against any given smart contract. A manual review which took up to 2 work days in 2018 can be performed in just a few minutes in 2021. Solidify’s capabilities and detection rates are continuously growing as our security engineers add new signatures and record associated risks.

How to Use Solidify

Let’s analyze a sample smart contract to illustrate the signature matching engine in action. Let’s look at ChainLink Token (LINK) and see how secure it is by running Solidify on its smart contract deployed at 0x514910771af9ca656af840dff83e8264ecf986ca:

% ./solidify -a 0x514910771af9ca656af840dff83e8264ecf986ca
2021/05/20 15:16:22 Loaded 5777 function signatures
2021/05/20 15:16:22 Processing 0x514910771af9ca656af840dff83e8264ecf986ca
2021/05/20 15:16:23 Loading signatures from: contracts/LINK_0x514910771af9ca656af840dff83e8264ecf986ca/LinkToken.sol
2021/05/20 15:16:23 Generating AST for contracts/LINK_0x514910771af9ca656af840dff83e8264ecf986ca/LinkToken.sol
2021/05/20 15:16:24 Installing solidity version 0.4.21
2021/05/20 15:16:28 Set solc version to 0.4.21
2021/05/20 15:16:29 Detected 19 signatures
2021/05/20 15:16:29 Storing report in reports/LINK_0x514910771af9ca656af840dff83e8264ecf986ca/LINK.yml
2021/05/20 15:16:29 Storing report in reports/LINK_0x514910771af9ca656af840dff83e8264ecf986ca/ChainLink Token (LINK) Security Memo.md

It took only a few seconds for Solidify to automatically download the contract source code, generate ASTs using an appropriate Solidity version, detect and match 19 unique function signatures. Below is a snippet of the Markdown report produced by the tool:

ERC-20 Security Assessment: ChainLink Token (LINK)

Auditor: Peter Kacherginsky
Solidify version: 1.2.3
Contract Source: 0x514910771af9ca656af840dff83e8264ecf986ca

Executive Summary

Date: 2021–05–20
Residual Security Risk Score: 2
Inherent Security Risk Score: 2

Risk Score: 2
Risk Description: External address call

Risk Score: 1
Risk Description: Use of assembly instructions
** Asset risks are calculated on the scale of 1–5 with 5 being the most severe.

No mitigations are necessary for this asset.

Risk Details

ChainLink Token has the following risks:

2 | External address call
The contract may invoke another function on a different smart contract in order to trigger functionality not defined within the contract itself

External address calls increase complexity and risk of the smart contracts. External calls are a prerequisite to re-entrantrancy vulnerabilities.

The following functions triggered this risk:

  • contractFallback(address,uint256,bytes memory)
  • transferAndCall(address,uint256,bytes memory)
  • transferAndCall(address,uint256,bytes memory)

1 | Use of assembly instructions
Assembly instructions use low level interface to directly manipulate the Ethereum Virtual Machine (EVM) as opposed to using the high level Solidity language. For example, inline assembly can make low level calls to other contracts, manipulate storage slots directly, and circumvent developer protections offered by higher level languages.

The use of assembly instructions increases complexity and obfuscates functionality of the smart contract while circumventing some of the inherent compiler protections.

The following functions triggered this risk:

  • isContract(address)

Mitigation Details

Implementation of the following risk mitigations drive the residual security risk score: N/A

Monitoring

Slack Monitoring: N/A
Slack and Pagerduty Monitoring: N/A

Matching Function Signatures

062afb9e84bbfded288268dcd3860be4fac7576697e563970dbfedd29dd9f5ff — add(uint256,uint256)
0f5d6c18af8bfe45551aea6c08ee36f2388d40721a399a99210754fb5b5d4fcc — isContract(address)
27711ded0a7898d7ac3feca9c539c7660909efcc5bf12c8e8b8612d522be6ac4 — contractFallback(address,uint256,bytes memory)
2d299e0f7d2ea990e5ca714c04fbac5ae68615d9404bf447f42234f28891fcd5 — transferFrom(address,address,uint256)
3ee09ec574744840001f024d57db0a57f4986733e71f8355bf8cd7af08e03ef4 — transferAndCall(address,uint256,bytes memory)
504201695c6fb08bebd3aaa9ccc252afd104cedddc5a5db8785ff1ec93e3255d -
[.. REDACTED ..]

Matching Function Sources:

signatures/openzeppelin/openzeppelin-contracts-1.1.0/contracts/token/BasicToken.sol
signatures/openzeppelin/openzeppelin-contracts-1.1.0/contracts/token/StandardToken.sol
signatures/openzeppelin/openzeppelin-contracts-1.10.0/contracts/math/SafeMath.sol
signatures/openzeppelin/openzeppelin-contracts-1.11.0/contracts/math/SafeMath.sol
signatures/openzeppelin/openzeppelin-contracts-1.3.0/contracts/math/SafeMath.sol
[.. REDACTED ..]

Most of the contract was taken from standard OpenZeppelin libraries or other known smart contracts which makes the function matching easy. There were only two informational findings related to the use of custom assembly code and making external function calls; however, those do not pose a security risk on their own. Based on the output above, the token would be rated with a risk score of (2) which is considered low risk from a security perspective.

Limitations

Solidify is purpose built to address our need to quickly and securely review ERC-20 and similar tokens. Having standardized token interfaces and a relatively limited functionality necessary to manage accounts is great for building signature databases which match a large percentage of contracts out there. But more complex assets such as AMMs and other DeFi applications require additional manual analysis because of the large percentage of custom code. However, Solidify is still beneficial for these applications when analyzing DeFi clones or for eliminating standard libraries from the manual review scope so analysts can focus on the custom logic.

Although Solidify is currently limited to Ethereum blockchain and Solidity, there are plans to expand the tool to other EVM-based platforms and support additional languages like Vyper.

Since all data in Solidify is manually populated by analysts there also exists a risk where we miss or misclassify features for a signature resulting in an incorrect assessment. We are concerned about deployed contracts trying to hide backdoors through malicious constructors, code obfuscation, exploiting the tool itself, and other techniques. Additional controls are built internally to validate signatures, detect anomalies and malicious behavior.

Future Development

We plan to open source Solidify later this year. In the meantime, additional development is focused on:

  • Improving accuracy of signature generation and detection logic
  • Integrating formal verification techniques to reduce the need for manual analysis

Conclusion

In this blog we have shared a novel tool and techniques used to automatically detect and classify smart contract risks. Solidify is a powerful and highly configurable tool that can be used by smart contract auditors, asset issuers, and other exchanges to help make digital asset ecosystems more secure. As the cryptoeconomy grows, tools like Solidify will become increasingly vital.

If you are interested in securing the future of finance, Coinbase is hiring.


Introducing Solidify — a tool to automatically detect and classify smart contract security risks was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Latam Insights: Paraguay Shuts Down Largest Illegal Bitcoin Mining Farm to Date, Argentina and El Salvador Discuss Bitcoin

Incident Post Mortem: May 19, 2021

By Bryant Khau and Leonardo Zizzamia

Summary

Between 5:50am and 7:38am PT on Wednesday, May 19th, there were connectivity issues with coinbase.com, Coinbase mobile apps, and Coinbase Pro. During this time, many users experienced slow load times and errors while attempting to access Coinbase, including features like buying, selling and trading. This post will detail the outage, explain what caused it, and describe the changes we’ve made to prevent similar failures going forward.

The Outage

There was a large spike of traffic due to many users reacting to a sudden price drop in the crypto market leading up to this incident (ETH dropped 20%, BTC dropped 25%). A group of oncall engineers convened after being paged for high error rates across several services.

The affected services were:

  • Logged out web servers: This caused users that weren’t logged-in to hit an error page when visiting coinbase.com.
  • GraphQL service: This caused parts of the mobile app to load very slowly and error ~10% of the time.
  • Coinbase Pro API: This caused Coinbase Pro to be partially unreachable.
  • Non-US card payment processing service: This caused non-US customers attempting to buy crypto with a card to be rejected.

Once these issues were identified, engineers split into different groups to investigate each issue in parallel and prioritize follow up actions.

Root Cause Analysis

In the days since the outage, we have reconstructed a clear picture of what happened since the first minute.

  1. The Logged out coinbase.com pages were largely unreachable as the instances started failing and took over 40 minutes to return to a healthy state. The rapid spike in requests ended up hitting a max threshold in Nginx router connections, which was manually increased during the incident. This ultimately addressed the bottleneck.
NodeJS HTML Response

2. We saw timeouts and increased latency on our GraphQL service, which aggregates data from underlying services. The timeouts were caused by GraphQL autoscaling up too slowly. The autoscaling eventually caught up and the errors subsided, restoring functionality to the mobile app and logged-in users.

GraphQL Errors

3. We saw that the database that powers the Coinbase Pro exchange had high latency and CPU load. Additionally the API servers that run our market data feed were under high CPU load. We increased the operation throughput configured on the database and also provisioned more API servers.

Coinbase Pro API Response Time

4. In our Non-US card payment processing service, the number of failed payments increased as the queue to process the payments became backlogged. We increased the number of queue workers and card payments started succeeding.

Queue Size

Improvements

At Coinbase, we’ve committed significant resources to improving our reliability, including regular load tests to prepare us for high periods of traffic. However, this incident has identified some blind spots to address, especially around very sudden spikes of traffic.

A common theme around several of the failures in this incident were autoscaling rules that weren’t tuned to the nature of traffic spikes that crypto markets can cause. We’re working on tailoring our load tests to better simulate real world situations, such as sudden traffic spikes. This will help surface more issues like untuned autoscaling rules, during controlled testing.

Another improvement that we are investing in is the implementation of kill switches for parts of the client application so that when failures happen, we can keep unaffected parts of our applications working while we work to address the failures.

We take the uptime and performance of our infrastructure very seriously, and we’re working hard to support the millions of customers that choose Coinbase to manage their cryptocurrency. If you’re interested in solving scaling challenges like those presented here, come work with us.


Incident Post Mortem: May 19, 2021 was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Latam Insights: Paraguay Shuts Down Largest Illegal Bitcoin Mining Farm to Date, Argentina and El Salvador Discuss Bitcoin

Hiring for the Coinbase infrastructure team

Hiring for the Coinbase Infrastructure team

A brief guide to how we scale our infrastructure talent

What makes great infrastructure?

The right cloud?
The right storage technology?
A great CDN?
The newest OS or kernel?
A shiny data center filled with blinkenlights?

At Coinbase, we don’t do infrastructure; we are Infrastructure.

We solve hard problems that sit in the little-known space between web-scale infrastructure and trading infrastructure. Those domains have historically relied on very different approaches to achieve reliability and scalability for customers. At Coinbase, we have to supersize both.

We’re working on multi-region, multi-cloud, reliability & scaling, decomposition of monoliths and a ton of other exciting challenges. For example, traditional infrastructure leans heavily on centralization and trust — however as Coinbase is growing the decentralized crypto ecosystem we have opportunities to rethink infrastructure in terms of decentralization and trustless architectures.

At the same time we understand that it takes teams, trust and collaboration to build a successful product. Coinbase’s success requires the best foundational infrastructure, therefore we deeply value our contributors who are building those foundations.

We work hard to make sure that our infrastructure team feels happy, engaged, successful and growing.

Why should I care?

Despite being a public company, the interesting challenges we are called on to solve are not completed; we’re just getting started!

As an engineer on the team, you will be responsible for defining, architecting and building the foundational services on which all products at Coinbase run. You’ll be part of a highly skilled team with an extremely collaborative and supportive culture. You will be learning from some of the top minds in the field and will be pushing yourself and the boundaries of infrastructure to its limits.

At Coinbase, we work hard to keep the rocketship in flight and we’re upgrading the engine to warp speed. All this while promoting collaboration and learning among our teammates. So if you’re feeling unchallenged at your current role, we can probably fix that — come talk to us.

How do you structure your teams?

We proactively split teams as they grow beyond 8–10 ICs. As of this post, we organize ourselves into the following teams:

Backend Platform: On the backend platform team you will deal with issues at Internet scale, maintaining datastores, APIs and our critical backend caching infrastructure supporting web and native applications. The team enables product teams to build highly available and scalable services that power much of Coinbase’s core service offerings.

Developer Productivity: The Developer Productivity team helps software engineers at Coinbase build and ship high-quality software. We create, secure, and scale first-class tooling and infrastructure that impacts nearly a thousand developers and all of Coinbase’s software systems. You will ensure our developers have a world-class development environment complete with test harnesses & frameworks, static analysis, and continuous integration. If you have a passion for helping other engineers iterate faster, and a monorepo environment with Bazel, Golang, and Ruby excites you, we want to hear from you.

Cloud: Are you an engineer that loves the low level details? Do you want to work on networking and cloud automation? Become a key part of our team by adding multi-region, multi-cloud to every part of Coinbase. As a Cloud engineer, you will create the building blocks that codify best practices for all engineers at Coinbase, upholding the highest standards of security, architecture, and reliability. Working with experts across the company, you are the key to maintaining infrastructure quality at Coinbase.

Reliability: Do you like bringing order to chaos in order to scale systems to 10x load? Can you see yourself being responsible for keeping a 24/7, multi-billion dollar market place running smoothly and reliably? SRE at Coinbase are responsible for helping engineers architect critical, high performance, high availability systems and making them run with minimal amounts of toil.

InfraSec: Do you want to be a part of the team that creates the foundations to protect the world’s cutting edge crypto-exchange? We’re looking for engineers that are excited about creating highly fortified, performant and easy to use infrastructure.

What do the current ICs on our infra team think?

Thomas: Senior Software Engineer

Working on the Infra team at Coinbase is like building the machine that builds the machine. With Coinbase’s global scale and quickly growing team, any change you make will have a big impact. No matter if you’re enabling other developers to better develop, build, deploy, or run their code, your work directly increases the productivity multiplier across the whole company.

Aarti: Senior Software Engineer

Infra is a foundational team at Coinbase composed of high performing individuals with diverse/complementary skill-sets and experiences who all come together as a team to solve hard scaling and reliability problems while being vigilant about security. At the core of the culture is a shared sense of purpose to improve Infrastructure and developer experience for the growing Coinbase Engineering teams.

Max: Software Engineer

The Infra team at Coinbase is building a secure and reliable foundation for a fast growing company. Working here enables you to gain a breadth of experiences ranging from designing scalable and secure network architecture that underlies thousands of production services running across multiple geographic regions to building essential tools that enable hundreds of engineers to bring their ideas to production with the least amount of friction.

Frances: Software Engineer

Infra builds the foundation of our entire system. We are in charge of building reliable and secure systems and tools, making them easy to use, and enabling engineers to keep their promises in production. Working here means that you get the best of both worlds — interacting with people across the company as well as getting immediate feedback on tooling from other engineers.

Come and join us as we scale the cryptoeconomy. Apply here for our Senior Backend Engineer role.


Hiring for the Coinbase infrastructure team was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Latam Insights: Paraguay Shuts Down Largest Illegal Bitcoin Mining Farm to Date, Argentina and El Salvador Discuss Bitcoin

Coinbase’s animated TabBar in React Native

By Jacob Thornton

Introduction to the “TabBar” user experience

You’ve probably used this interaction countless times in your day to day life and not spent much time thinking about it. You see it on things like Instagram, Twitter, Apple Music, AND most recently on the Coinbase prices screen.

It’s a simple tab component, with a scroll-away Header and TabBar that pins to the top of the screen. It allows you to swipe through the different TabViews, and it treats the overall scroll position intelligently (i.e. if you scroll past the Header on one TabView, switching tabs keeps the global header in the same position, irrespective of tab scroll).

However, despite how universal this UX experience has become, behind the scenes it still requires a large lift of complex gestures and state management to get it feeling and performing “just right” for the end user.

At Coinbase we were able to pull this off for our recent React Native rewrite and wanted to share a little bit more about the UX details that went into this interaction, as well as the code needed to make it happen.

Glossary of UX terms

Before we get started, let’s define some terminology that will help us describe the core components needed for pulling off this experience. We also extracted a simple companion example project — which you can use to follow along or help build your own working version of this experience.

  1. Header This is the top typographic lockup on the Screen.
  2. TabBar This is a list of tab buttons. It highlights the active tab and allows you to navigate between each TabView with a tap or swipe.
  3. TabView This is the scrollable tab content. This can be asset cells, tweets, photos, or any other content.
  4. NavBar This is the top screen navigation. You often see this with back arrows or other icons up here as well. But to keep it simple we just have a title.
  5. Screen — The top level container component (root level React component).

Beyond components we will also be referring to the following terms:

  1. Interpolation — Maps input ranges to output ranges, typically using a linear interpolation but also supports easing functions.
  2. FPS — Frames per second.
  3. Bridge Traffic — Messages being sent over React Native’s bridge between its javascript and native layer.
  4. Active TabView — The currently visible TabView.
  5. Inactive TabView — TabViews that are off screen (not currently visible).

Guide to interactions

There are a handful of key interactions that go into making the TabView look and feel organic. Many of these behaviors were identified through studying external implementations like Twitter and Instagram, as well as platform specific applications and specifications like Apple’s HIG and Google’s Material Design.

Here is a list of key interactions (all of which you will find in our sample project on github). Study the gif above to see if you can pick out each of these subtle behaviors.

  1. The NavBar should fade in/out with a timer based animation when the scroll reaches a certain position.
    Many people make the mistake of interpolating the NavBar title opacity based on the page’s scroll position. While this works really well when scrolling slowly, if you scroll more quickly (standard in day-to-day use of applications) you’ll notice the NavBar title will often transition in too abruptly.
    We first noticed this behavior when analyzing Apple Note’s UI, but you’ll see in all of Apple’s native applications (Apple Music, Mail, etc.) that they implement this same fade trigger point.
    The ideal trigger point for this animation is right as the Header title is obstructed by the NavBar.
  2. The Header should slide up and off the screen (beneath the NavBar) on scroll.
    This might seem obvious, but getting this right on React Native can be quite challenging. One important limitation of React Native is that there are no nested scroll surfaces. This has major consequences:
    a. Because TabViews must be independently scrollable (with their own independent scroll positions), the top-level Screen container cannot be scrollable.
    b. Because the top-level Screen container cannot be scrollable, we must simulate the effect that the top of the page is “scrolling” by interpolating the active TabView’s scroll position to a translateY offset for the Header. This transform gives the visual appearance of the Header scrolling with the TabBar and TabViews up and off the page, without actually scrolling the page (more on managing scroll position, and the complexity that comes with that in the implementation details below).
  3. The TabBar should slide up and then pin to the bottom edge of the NavBar.
    Similar to the Header, the TabBar must be slid up the screen using translateY. Unlike with Header, we need to cap the total distance the component can travel up the screen, to give the effect that the tabs are pinned beneath the NavBar.
  4. Once the TabBar is pinned, a bottom border should be faded in to add depth to TabViews sliding beneath it.
    Unlike with the NavBar, the bottom border opacity should be interpolated in direct correlation to the scroll position. The reason we do this is to guarantee that the border is always opaque whenever TabView is passing under TabBar.
  5. TabView should scroll up and pin directly beneath the TabBar, ensuring that anytime you swipe or navigate between TabViews, the header position is always correctly maintained.
    Like TabBar, Tabview should appear visually pinned beneath TabBar. Once pinned, the TabView content should appear to scroll beneath the pinned TabBar.

Implementation

Mobile users are accustomed to smooth and well-designed interfaces that quickly respond to their interactions and provide prompt visual feedback. At Coinbase we use custom tooling to track and measure our fps, aiming to keep key interactions like navigation, gestures, etc. at 60fps. In order to do this in React Native, it’s imperative that we both limit bridge traffic (reducing JS-driven animations) and reduce render cycles (minimally update state/context api).

Performance concerns, coupled with React Native’s limitation around no nested scroll surfaces, means our best option for powering the above interactions is to manage a single, shared Animated.Value (that we call ScrollY).

To populate our Animated.Value we use React Native’s event method and native driver, adding the below snippet to TabView’s onScroll event.

https://medium.com/media/2a0ae563db797cc8dc5149aae7d4805c/href

Note: We update the scrollY event here based on whether a given tab is active. This makes sure there is only ever a single TabView updating scroll positions at any given time.

Now that we have ScrollY management locked in, we can begin to interpolate its value across our different components to achieve the interactions covered above.

  1. The NavBar title should fade in/out with a timer based animation when the scroll reaches a certain position.

Here we set a listener for our scrollY value change. When this value changes, if the scroll distance covered is 60% of the Header (meaning the header has traveled upward beneath the NavBar and 60% of its height is covered by the NavBar), we trigger a 300ms native opacity animation for the NavBar title. Because our Navbar is fixed to the top of the Screen, we don’t have to worry about any complex positioning or offset logic.

https://medium.com/media/b03bdaa8cabefc5d3d1e071fbf119450/href

2. The Header should slide up and off the screen (beneath the NavBar) on scroll.

The React Native interpolation API takes a value that changes over time and interpolates it based on a range of inputs (e.g. 0 to 1) and outputs it to a given range (e.g. 0 to 100). So for example, if we had an Animated.Value updated to 0.3 and ran through the below interpolation, the interpolated output would be 30.

https://medium.com/media/82e9c78f3b8d758d1bfe17655a4583c2/href

With this in mind, let’s consider our Header interpolation.

https://medium.com/media/40670c20d52998e73fab464312887371/href

Here we are interpolating a ScrollY value that should theoretically have an input range of 0 (top of scroll) and our Header height (let’s say 80pt).

There is a bit of nuance here however because of how iOS treats 3 scenarios: pull to refresh, rubberbanding, and overscroll.

Inside our TabView we provide special contentInset and contentOffset patterns for iOS. This allows our “pull to refresh” spinner to correctly pull down beneath our Header (visually at the top of TabView).

Note: We are able to achieve the same affect in a simpler manner on Android by simply applying a paddingTop to our contentContainers style. We are able to get away with this on Android because it doesn’t have overscroll.

https://medium.com/media/d1fdb79252cfd913525359a88dec4465/href

When returning to our Header interpolation, the inputRange used must be different for iOS (to account for the contentOffset). Specifically:

https://medium.com/media/57a13574e2a8e8df33fb7aafe5a4ba19/href

The final thing to note here is that we are interpolating those input values (ScrollOffset to ScrollOffset + Header) to an output range of (0 to negative Header height). And we’re using React’s interpolate “clamp” value to prevent the output value from exceeding outputRange (whereas the default is “extend” which will exceed our outputRange).

The result is that as you scroll TabView the header will be transformed up and off the screen until it is completely hidden (negative Header), and no further.

https://medium.com/media/764cfa006bcc09fb9a7601e0360ee188/href

3. The TabBar should slide up and then pin to the bottom edge of the NavBar.

By default, the TabBar is rendered inside of the react-native-tab-view’s TabView component using { position absolute, top: 0 }, which effectively gives us the visual “pinning” effect.

The trick here is we’re using interpolation to first push the TabView further down the screen then it’s natural render position (unlike we did with our Header component), and then using ScrollY change events to transform it back to it’s natural 0 placement. This is why our output range in the below invocation sets an initial transform of the height of the Header, and translates it up into it’s pinned final pinned position of 0.

https://medium.com/media/60d0d00f7400cb84776e4d26bd26af0e/href

4. Once the TabBar is pinned, a bottom border should be faded in to add depth to TabViews sliding beneath it.

To fade the border in, we set an input range from the top of the Header to the Header + TabBar Height. This ensures that the border doesn’t begin fading in until the header has been scrolled out of view, and that it is completely faded in by the time the TabView children scroll beneath it.

We then interpolate the value to an outputRange of 0 to 1, which is the value range for the opacity property in React Native.

https://medium.com/media/e5f1cf232240e9f45bbeb327fe3ba123/href

5. TabView should slide up and pin directly beneath the TabBar, ensuring that anytime you swipe or navigate between TabViews, the header height is always correctly maintained

This is arguably the most difficult piece of this entire architecture. Relying on a single ScrollY value greatly simplifies the above interpolations, however it becomes problematic as soon as you begin to take into account inactive TabViews. If you aren’t doing something to proactively synchronize inactive scroll views, then navigating between TabViews will display scroll positions at incorrect placements (as the inactive TabViews won’t have compensated for the Header being shown or hidden).

To solve this we must populate the following 4 variables.

  1. index The active tab index, provided by our Tab library react-native-tab view.
https://medium.com/media/642e6f2a681f797c1779a5085d35d225/href

2. tabKey The tab identifier, provided by our Tab library react-native-tab-view and passed down to individual TabView components.

https://medium.com/media/f617a88b0aba8acff3161ae5865be73d/href

3. tabkeyToScrollableChildRef This is a map of tab identifiers to scrollable component refs (TabViews). It will be used later to passively update all inactive TabView scroll offsets.

https://medium.com/media/11a73d9580498507536a79eab592a651/href

4. tabkeyToScrollPosition This is a map of tab identifiers to their local scroll position. It will be used to compare cached scrollPositions to Header offsets to determine if inactive TabViews scroll offsets need to be adjusted.

https://medium.com/media/09b36ccb9b0f66df76cf07a462eff654/href

To begin synchronizing scroll offsets, we first start by populating tabkeyToScrollableChildRef using React’s special ref prop. When you pass a method to the ref prop in React, the incoming method is invoked with a reference to the component. We then take this reference, along with a TabView’s local TabKey reference (passed in by react-native-tab-view), and call up to our trackRef method.

https://medium.com/media/5c94cbbc7561497f9ade1ac3470d606a/href

Next we populate our tabKeyToScrollPosition map by listening in on our global ScrollY value and inferring the activeTab by mapping index to our tabs array. This allows us to know where all of our tabs are scrolled (even inactive tabs) at any given time.

https://medium.com/media/50b937eb46dfe94777413c2be450d5a1/href

The final step to synchronize scroll offsets is to make use of onMomentumScrollEnd and onScrollEndDrag events to invoke a custom method we call syncScrollOffset.

SyncScrollOffset iterates through inactive TabKeys and updates their scroll offsets depending on if the Header is scrolled in or out of view. Doing so will ensure that anytime you swipe or navigate between TabViews, the header height is always correctly maintained.

https://medium.com/media/68bd2a308c4653801afcb118ef7e920c/href

Summary

At Coinbase, we’re spending a lot of time getting these interactions as close to perfect as possible. An important part about our transition away from Native technologies to React Native has been about not only maintaining our quality bar, but raising it. This happens through deep UX investments, our multi-platform design system, and more.

If this type of work interests you, or if you have any questions about the above implementation, please don’t hesitate to reach out and check out our job board — we’re hiring!

And check out our previous posts on our React Native journey such as Onboarding thousands of users with React Native and Optimizing React Native.

This website contains links to third-party websites or other content for information purposes only (“Third-Party Sites”). The Third-Party Sites are not under the control of Coinbase, Inc., and its affiliates (“Coinbase”), and Coinbase is not responsible for the content of any Third-Party Site, including without limitation any link contained in a Third-Party Site, or any changes or updates to a Third-Party Site. Coinbase is not responsible for webcasting or any other form of transmission received from any Third-Party Site. Coinbase is providing these links to you only as a convenience, and the inclusion of any link does not imply endorsement, approval or recommendation by Coinbase of the site or any association with its operators.

All images provided herein are by Coinbase.


Coinbase’s animated TabBar in React Native was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Latam Insights: Paraguay Shuts Down Largest Illegal Bitcoin Mining Farm to Date, Argentina and El Salvador Discuss Bitcoin