Anaconda and Accelerating AI Development with Rob Futrick

2 maj 2024 | 44 min

Spring AI and Java in 2024

5 april 2024 | min

Java is one of the most widely used programming languages, and a key contributor to its success is VMware Tanzu’s Spring, the most common framework for Java development. The Spring Framework is built on top of the Java Virtual Machine and provides a consistent programming and configuration model for application developers. From inception, it was designed with developer experience and modularity in mind.

The open-source application framework has been accelerating Java development times since its inception in 2004 (Happy 20th birthday). Since then, the platform has been expanding: growing 50% year over year during the last five years. In this blog we’re exploring what makes Spring important to Java, how the framework influenced the developer experience, and a look at the latest version of Spring, which introduces features to support AI integration.

Spring Success

Two decades ago, there were dozens of different ways to connect to a database, something just about every application has to do at some point. “At the time, the various approaches were very cumbersome: developers wrote a lot of code and gained very little functionality,” explained Mark Pollack, a Senior Staff Engineer, Tanzu Division, Broadcom. “Spring provided a lot of value by simplifying the process. Just getting a single app to talk to a database, present a web form, and do transactions correctly was a huge win. In that era, developers could spend weeks trying to create that function.”

Another reason for its success is its enterprise-focus. Most open source projects concentrate on the consumer market. However, large companies invest a lot of money building applications to run their businesses. “At the end of the day, large corporations’ largest expense is probably their developers,” explained Ryan Morgan, Senior Director of Engineering, Tanzu Division, Broadcom. Spring makes developer teams more efficient, which greatly enhances the bottom line.

Through the years, the ecosystem has grown. “There’s a large and vibrant community behind Spring,” noted Tanzu’s Morgan. Now, it has more than 200 different technology starters. These software building blocks make it simpler for software engineers to integrate their code with different third-party systems.

Get a Great Start

Development problems evolve over time, so various elements were added. Spring Initializer is a bootstrapping tool, a way for developers to create a new project. “Normally, software engineers started from a blank piece of paper and had to figure out what type of project it was and what type of libraries were needed,” said Tanzu’s Morgan. Then, they searched the web to find some place in the documentation that told them what library dependency was needed to add for different pieces of functionality. Then invariably, you cut and pasted from something that wasn’t consistent. You ended up with a mess.”

With Spring Initializer, software engineers go to a website which has clear instructions about what the options are and presents them in a typical web form. Then, they enter the Generate command and out comes a shell that they can use to start building their program. The solution does not generate any code but solves the problem of finding the right application dependencies. Developers start faster and are less frustrated than previous methods.

Under development is Spring CLI, which not only creates the shell of the app but also includes code. The advances have a significant impact because 1 million new projects are created each month.

Meet the Need for Development Speed

The last 10 years have seen a major move to container deployment and Spring has aligned with this paradigm shift. “Really, when you think about all those cloud native patterns, a lot of those container functions are really baked into our projects already,” stated Tanzu’s Morgan. “If you want to do distributed configuration, we have a solution for that. You want to do service discovery; Spring has patterns and tools for that.”

Recently, a major change to Spring occurred. Rather than release new functions autonomously, they are gathered and bundled into Spring Boot. Version 3.0, which is based on Spring Framework 6.0, requires Java 17 or above. Previously, Spring supported Java 8, so the change is significant for some companies.

Better performance is one benefit from the change. “We’ve seen customers realize 15% performance improvements, just from doing the upgrade,” said Tanzu’s Morgan.

Java Supports Generative AI

AI is being woven into many applications, especially with the emergence of Generative AI solutions. They represent a quantum leap in capabilities and overall intelligence compared to previous iterations of AI. One reason why today’s generative models are gaining so much attention is that they work with much larger volumes of information (hundreds of billions of words) and larger data models (hundreds of billions of parameters) compared to previous AI systems. They possess impressive and unprecedented power. Consequently, they can perform very sophisticated functions.

However as developers try to take advantage of the functionality, platform diversity again presents development challenges. “OpenAI has their API, Amazon Bedrock offers a different one, and so do other companies,” noted Tanzu’s Pollack.

A guiding focus and design principle in the Spring framework is simplifying such work by providing common abstractions over similar technologies and interfaces. Spring AI is quickly becoming the starting point when Java developers write AI applications. “Spring AI has the common patterns that Spring developers are used to,” noted Tanzu’s Pollack. It can abstract out models, clients, etc. in ways that are familiar to Spring users.”

Another crucial part of AI applications is using a vector database. Spring supports multiple vector databases, and its portable API simplifies changing implementations. So, Spring streamlines AI application development.

Java has been a popular programming language for enterprises for decades. Spring provides software engineers with tools that help them enhance the development process. The framework has reached its 20th year of empowering developers, and its years, engaged community is laying the groundwork for continued expansion in the coming decades. “Maybe one reason why Spring continues to do well is it constantly tries to improve itself and doesn’t just rest on its laurels,” concluded Tanzu’s Pollack.

The post Spring AI and Java in 2024 appeared first on Software Engineering Daily.

Vercel AI with Lee Robinson

30 januari 2024 | 49 min

Algolia with Sean Mullaney

23 januari 2024 | 49 min

JetBrains AI with Jodie Burchell

16 januari 2024 | 55 min

AWS re:Invent Special: Sagemaker with Ankur Mehrotra

9 januari 2024 | 35 min

Pinecone Vector Database with Marek Galovic

2 januari 2024 | 39 min

Vespa.ai with Jon Bratseth

26 december 2023 | 46 min

GitHub Copilot with Joseph Katsioloudes

19 december 2023 | 50 min

Privacy in the Age of Generative AI

18 december 2023 | min

On a recent trip to my hometown in Eastern Canada, my father picked me up at the airport. One of the first things he asked me was, “Is AI going to take everyone’s jobs?”.

When AI, generative AI, and large language models (LLM) have become topics of conversation within the senior citizen community of rural Canada, you know it’s on everyone’s minds. Generative AI, and especially the use of LLMs, is the “new new thing”. It dominates my X (i.e. Twitter) feed and nearly every conversation I have about technology.

There’s justifiably a ton of excitement about the power of generative AI, reminiscent of the introduction of the Internet or the first smartphone. Generative AI is poised to transform how we build products, design drugs, write content, and interact with technology. But as the utilization of AI grows, many governments and companies have raised concerns about the privacy and compliance issues that adopters of these technologies face.

The core challenge posed by generative AI right now is that unlike conventional applications, LLMs have no “delete” button. There’s no straightforward mechanism to “unlearn” specific information, no equivalent to deleting a row in your database’s user table. In a world where the “right to be forgotten” is central to many privacy regulations, using LLMs presents some difficult challenges.

So what does all this mean for businesses that are building new AI-powered applications or AI models?

In this post, we’ll explore this question and attempt to provide answers. We’ll examine the potential impact of generative AI, ongoing compliance hurdles, and a variety of privacy strategies. Finally, we’ll examine a novel approach grounded in the IEEE’s recommended architecture for securely storing, managing, and utilizing sensitive customer PII (Personally Identifiable Information)—the data privacy vault.

Generative AI’s privacy and compliance challenges

Imagine the following scenario: You’ve just copied and pasted sensitive contract details into an LLM to get some quick assistance with routine contract due diligence. The LLM serves its purpose, but here’s the catch: depending on how it’s configured, that confidential contract data might linger within the LLM, accessible to other users. Deleting it isn’t an option, predicting its future use—or misuse—becomes a daunting task, and retraining the LLM to “roll it back” to its state before you shared those sensitive contract details can be prohibitively expensive.

The only foolproof solution?

Keep sensitive data far away from LLMs.

Sensitive information, including internal company project names, core intellectual property, or personal data like birthdates, social security numbers, and healthcare records, can inadvertently find its way into LLMs in several ways:

Training data: LLMs are trained and honed on expansive datasets that often contain PII. Without robust anonymization or redaction measures in place, sensitive data becomes part of the model’s training dataset, meaning that this data can potentially resurface later.
Inference: LLMs generate text based on user inputs or prompts. Much like training data, a prompt containing sensitive data seeps into the model and can influence the generated content, potentially exposing this data.

Privacy laws

AI data privacy is a formidable challenge for any company interested in investing in generative AI technology. Recent temporary bans of ChatGPT in Italy and by companies like Samsung have pushed these concerns to the forefront for businesses looking to invest in generative AI.

Even outside of generative AI, there are increasing concerns about protecting data privacy. Meta was recently fined $1.3 billion by the European Union (EU) for its non-compliant transfers of sensitive data to the U.S. And this isn’t just an issue for companies doing business in the EU.

There are now more than 100 countries with some form of privacy regulation in place. Each country’s privacy regulations include unique and nuanced requirements that place a variety of restrictions on the use and handling of sensitive data. The most common restrictions relate to cross-border data transfers, where sensitive data can be stored, and to individual data subject rights such as the “right to be forgotten.”

One of the biggest shortcomings of LLMs is their inability to selectively delete or “unlearn” specific data points, such as an individual’s name or date of birth. This limitation presents significant risks for businesses leveraging these systems.

For example, privacy regulations in Europe, Argentina, and the Philippines (just to name a few) all support an individual’s “right to be forgotten.” This grants individuals the right to have their personal information removed or erased from a system. Without an LLM delete button, there’s no way for a business to address such a request without retraining their LLM from scratch.

Consider the European Union’s General Data Protection Regulation (GDPR), which grants individuals the right to access, rectify, and erase their personal data—a task that becomes daunting if that data is embedded within an LLM. GDPR also empowers individuals with the right to object to automated decision-making, further complicating compliance for companies that use LLMs.

Data localization requirements pose another challenge for users of LLMs. These requirements pertain to the physical location where customer data is stored. Different countries and regions have precise laws dictating how customer data should be handled, processed, stored, and safeguarded. This poses a significant challenge when using an LLM used for a company’s global customer base.

Data Subject Access Requests (DSARs) under GDPR and other laws add another layer of complexity. In the EU and California, individuals (i.e., “data subjects”) have the right to request access to their personal data, but complying with such requests proves challenging if that data has been processed by LLMs.

Considering the intricate privacy and compliance landscape and the complexity of LLMs, the most practical approach to maintaining compliance is to prevent sensitive data from entering the model altogether. By implementing stringent data handling practices, businesses can mitigate the privacy risks associated with LLMs, while also maintaining the utility of the model. Many companies have already decided that the risks are too high, so they’ve banned the use of ChatGPT, but this approach is shortsighted. Properly managed, these models can create a lot of value.

Privacy approaches for generative AI

To address the privacy challenges associated with generative AI models, there have been a few proposals such as banning or controlling access, using synthetic data instead of real data, and running private LLMs.

Banning ChatGPT and other generative AI systems isn’t an effective long-term strategy, and these other “band aid” approaches are bound to fail as people can find easy workarounds. Using synthetic data replaces sensitive information with similar-looking but non-sensitive data and keeps PII out of the model, but at the cost of losing the value that motivated you to share sensitive data with the LLM in the first place. The model loses context, and there’s no referential integrity between the synthetically generated data and the original sensitive information.

The most popular approach to addressing AI data privacy, and the one that’s being promoted by cloud providers like Google, Microsoft, AWS, and Snowflake, is to run your LLM privately on their infrastructure.

For example, with Snowflake’s Snowpark Model Registry, you can take an open source LLM and run it within a container service in your Snowflake account. They state that this allows you to train the LLM using your proprietary data.

Snowpark Model Registry and Container Service (Source: Snowflake Blog)

However, there are several drawbacks to using this approach.

Outside of privacy concerns, if you’re choosing to run an LLM privately rather than take advantage of an existing managed service, then you’re stuck with managing the updates, and possibly the infrastructure, yourself. It’s also going to be much more expensive to run an LLM privately. Taken together, these drawbacks mean running a private LLM likely doesn’t make sense for most companies.

But the bigger issue is that, from a privacy standpoint, private LLMs simply don’t provide effective data privacy. Private LLMs give you model isolation, but they don’t provide data governance in the form of fine-grained access controls: any user who can access the private LLM can access all of the data that it contains. Data privacy is about giving a user control over their data, but private LLMs still suffer from all of the intrinsic limitations around data deletion that are blocking the adoption of public LLMs.

What matters to a business—and individual data subjects—is who sees what, when, where, and for how long. Using a private LLM doesn’t give you the ability to make sure that Susie in accounting sees one type of LLM response based on her job title while Bob in customer support sees something else.

So how can we prevent PII and other sensitive data from entering an LLM, but also support data governance so we can control who can see what and support the need to delete sensitive data?

A new approach to PII management

In the world of traditional data management, an increasingly popular approach to protecting the privacy of sensitive data is through the use of a data privacy vault. A data privacy vault isolates, protects, and governs sensitive customer data while facilitating region-specific compliance with laws like GDPR through data localization.

With a vault architecture, sensitive data is stored in your vault, isolated outside of your existing systems. Isolation helps ensure the integrity and security of sensitive data, and simplifies the regionalization of this data. De-identified data that serve as references to the sensitive data are stored in traditional cloud storage and downstream services.

De-identification happens through a tokenization process. This is not the same as LLM tokenization, that has to do with splitting texts into smaller units. With data de-identification, tokenization is a non-algorithmic approach to data obfuscation that swaps sensitive data for tokens. A token is a pointer that lets you reference something somewhere else while providing obfuscation.Traditional data management versus a data privacy vault architec

*Traditional data management versus a data privacy vault architecture*

Let’s look at a simple example. In the workflow below a phone number is collected by a front end application. The phone number, along with any other PII, is stored securely in the vault, which is isolated outside of your company’s existing infrastructure. In exchange, the vault generates a de-identified representation of the phone number (e.g. ABC123). The de-identified (or tokenized) data has no mathematical connection with the original data, so it can’t be reverse engineered.

Any downstream services—application databases, data warehouse, analytics, any logs, etc.—store only a token representation of the data, and are removed from the scope of compliance:

Example of a data privacy vault in action

Additionally, a data privacy vault can store sensitive data in a specific geographic location, and tightly control access to this data. Other systems, including LLMs, only have access to non-sensitive de-identified data.

The vault not only stores and generates de-identified data, but it tightly controls access to sensitive data through a zero trust model where no user account or process has access to data unless it’s granted by explicit access control policies. These policies are built from the bottom, granting access to specific columns and rows of PII. This allows you to control who sees what, when, where, for how long, and in what format.

For example, let’s say we have a vault containing customer records with columns defined for a customer’s name, social security number (SSN), date of birth (DOB), and email. In our application we want to support two types of users: support and marketing.

Support doesn’t need to know the exact details about a customer, they only need masked data so they can speak to the customer by name and verify their identity using the last four digits of the customer’s SSN. We can create a policy for the role support that grants access to only the limited view of the data.

ALLOW READ ON users.full_name, users.ssn, users.email WITH REDACTION = MASKED
ALLOW READ ON users.dob WITH REDACTION = REDACTED

Similarly, a marketing person needs someone’s name and email, but they don’t need the customer’s SSN or need to know how old someone is.

ALLOW READ ON users.full_name, users.email WITH REDACTION = PLAIN_TEXT
ALLOW READ ON users.dob WITH REDACTION = MASKED
ALLOW READ ON users.ssn WITH REDACTION = REDACTED

With roles and policies similar to ones above in place, the same de-identified data is exchanged with the vault. Based on the role and associated access control policies for the caller, different views of the same sensitive data can be supported.

*Different views of sensitive data based on role.*

A privacy firewall for LLMs

Companies can address privacy and compliance concerns with LLMs with a similar application of the data privacy vault architectural pattern. A data privacy vault prevents the leakage of sensitive data into LLMs, addressing privacy concerns around LLM training and inference.

Because data privacy vaults use modern privacy-enhancing technologies like polymorphic encryption and tokenization, sensitive data can be de-identified in a way that preserves referential integrity. This means that responses from an LLM containing de-identified data can be re-identified based on zero trust policies defined in the vault that let you make sure that only the right information is shared with the LLM user. This lets you make sure Susie in accounting only sees what she should have access to (i.e., account numbers and invoice amounts) while Bob in customer support sees only what he needs to do his job.

Preserving privacy during model training

To preserve privacy during model training, the data privacy vault sits at the head of your training pipeline. Training data that might include sensitive and non-sensitive data goes to the data privacy vault first. The vault detects the sensitive data, stores it within the vault, and replaces it with de-identified data. The resulting dataset is de-identified and safe to share with an LLM.

*Model Training Pipeline with a Data Privacy Vault*

An LLM doesn’t care whether my name, Sean Falconer, is part of the training data or some consistently generated representation of my name (such as “dak5lhf9w”) is part of the training data. Eventually, it’s just a vector.

Preserving privacy during inference

Sensitive data may also enter a model during inference. In the example below, a prompt is created asking for a summary of a will. The vault detects the sensitive information, de-identifies it, and shares a non-sensitive version of the prompt with the LLM.

Since the LLM was trained on non-sensitive and de-identified data, inference can be carried out as normal.

*Example of de-identifying inference data with an LLM and a data privacy vault*

On egress from the LLM, the response is passed through the data privacy vault for re-identification. Any de-identified data will be re-identified assuming the end-user has the right to see the information, according to explicit access control policies configured in the vault.

Privacy and compliance

From a privacy and compliance standpoint, using a data privacy vault means that no sensitive data is ever shared with an LLM, so it remains outside of the scope of compliance. Data residency, DSARs, and delete requests are now the responsibility of a data privacy vault that’s designed to handle these requirements and workflows.

Incorporating the vault into the model training and inference pipelines allows you to combine the best of modern sensitive data management with any LLM stack, private, public, or proprietary.

Final thoughts

As every company gradually morphs into an AI company, it’s critically important to face data privacy challenges head-on. Without a concrete solution to data privacy requirements, businesses risk remaining stuck indefinitely in the “demo” or “proof-of-concept” phase. The fusion of data privacy vaults and generative AI offers a promising path forward, freeing businesses to harness the power of AI without compromising on privacy.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.

The post Privacy in the Age of Generative AI appeared first on Software Engineering Daily.

Weights & Biases with Chris Van Pelt

12 december 2023 | 38 min

De-scoping Your AWS Services from Data Residency Requirements

11 december 2023 | min

From the widely recognized GDPR in Europe to Brazil’s LGPD regulations, and the more recent introduction of India’s DPDP law, over 100 countries now have some form of privacy regulation in place. What’s common among many of these regulations is the concept of data residency – the physical location of your data. However, each region’s requirements bring their own unique nuances, encompassing restrictions on data transfer, data storage locations, and individual data rights.

Navigating this complex sphere of privacy regulations is a huge burden for many companies born in the cloud. Their data simply ends up everywhere, and tracking down the locations, adhering to local laws, and even storing and using it locally is enormously complex and expensive.

Over the past year, I’ve engaged with numerous companies eager to expand their businesses into new markets, such as Europe and Australia. However, they’ve encountered a significant roadblock – the absence of a robust technology solution to address the data residency requirements of these regions. As a result, they face the expensive and nightmarish scenario of duplicating their cloud infrastructure for each new region, which not only hampers operational efficiency but also limits their data analyst and scientists from running analytics globally.

In this blog post, I offer a solution to this pressing technology and business challenge by introducing a PII data privacy vault. This architectural approach to data privacy effectively removes the burden of data residency, compliance, and data security responsibilities from your infrastructure, providing a seamless path for global expansion and data management.

Let’s dive in.

Data Residency and Barriers to Expansion

To grasp the intricacies of regulatory compliance in the context of global expansion, it’s important to understand a few key concepts.

Compliance

Compliance denotes a business’s adherence to the laws and regulations governing data privacy and protection. These regulations are contingent on the geographic location of the customer whose data is being collected. Ensuring compliance is imperative for legal reasons as it shields businesses from financial penalties, license revocations, and the erosion of customer trust.

Data Residency

Data residency pertains to the physical location where customer data is stored. For instance, a website may serve customers in the EU, but their data could be hosted on a server located in Chicago. Different countries and regions have precise laws dictating how customer data should be handled, processed, stored, and safeguarded, making data residency a critical consideration.

Varying Regulations

The complexity surrounding data residency and compliance obligations primarily arises from the diversity of regulations worldwide. For instance, the European Union (EU) has GDPR, Brazil follows LGPD, and the United States enforces a patchwork of state-specific laws like CCPA in California and CTDPA in Connecticut. These regulations diverge significantly in terms of their stipulations and penalties.

Barriers to Global Expansion

The disparities in regulations and compliance requirements often pose formidable obstacles for companies striving to attain a global presence. Navigating diverse regulatory frameworks demands significant time, resources, and expertise. The resulting complexity frequently dissuades businesses from venturing into new markets, thereby constraining opportunities for global expansion.

We’ve looked at the problem, now, let’s explore an approach to addressing these challenges.

What is a Data Privacy Vault?

A data privacy vault isolates, protects, and governs access to sensitive customer data. Within the vault, confidential information is securely stored, while abstract and non-sensitive tokens, serving as references, are retained in conventional cloud storage. This means that only non-sensitive tokenized data is accessible to other systems, ensuring the utmost protection and compliance.

In a recent IEEE article, the authors made a case that this architectural approach to data privacy is the future of privacy engineering. Just as any modern system likely contains back end services, a database, and a warehouse, all modern systems need a data privacy vault to safely store, handle, and use of sensitive customer PII.

Traditional PII management versus a data privacy vault (source: IEEE).

Let’s take a look at a specific example for a simple web application. In the image below, a phone number is being collected by a front-end application. For effective de-scoping, it’s ideal to initiate the de-identification process at the earliest stage in the data lifecycle. In this scenario, the phone number is stored directly within the vault during collection at the front end.

Example of vault architecture for collecting sensitive customer PII.

Within the vault, the phone number, alongside any other personally identifiable information (PII), is stored within a robust and isolated environment, segregated from your organization’s existing infrastructure. All downstream services, ranging from application databases to data warehouses, analytics platforms, and logging systems, interact solely with tokenized (de-identified) representations of the data. Queries against the PII for specialized operations or algorithmic operations against PII execute directly within the vault.

Access to de-tokenize or re-identify data is controlled through a zero trust model. Policy-based rules control who sees what, when, where, and for how long on a row and column level.

Controlling access to vault data based on who is requesting the data.

The vault combines the principle of isolation, zero trust, privacy-enhancing technologies, and governance controls to insulate your systems from ever having to touch PII directly. This places your AWS components beyond the scope of regulatory compliance, assuring a higher level of data protection and adherence to data residency requirements.

Your AWS Services Handle Only De-identified Data

Let’s assume we have a simple application infrastructure as shown below with AWS Amplify providing the web server infrastructure, DynamoDB for application storage, and Redshift for warehousing.

Example web application infrastructure running on AWS.

Without a vault in place, everything within our AWS account is under compliance and security scope.

By introducing the vault as shown below (in this example, the collection of PII is handled directly from the vault), we de-scope all our AWS services. The services are only ever handling de-identified data, including the warehouse.

Many analytical operations can be performed with de-identified data provided the data is consistently generated. A warehouse doesn’t need to have access to someone’s name, it only needs a consistently generated representation of the name in order to execute counts, group bys, and joins.

Example of de-scoping AWS services with a data privacy vault.

Storing PII to Different Regionalized Vaults

With Skyflow, a data privacy vault company, you can host vaults in various global regions and route sensitive data to a specific regional vault for storage and use. For instance, consider how the following application architecture meets data residency requirements across multiple regions:

Using regional multiple vaults to comply with data residency requirements.

Your company’s site collects customer PII during account creation.
On the client side, the website detects the customer’s location.
Detecting that the customer is in the EU, the client-side code uses Skyflow’s SDK to collect the PII data and store it in your company’s data privacy vault in Frankfurt, Germany. Note: For customers based in the US, the PII data is instead routed to the data privacy vault in the US (in this case, Virginia).
The EU-based customer’s sensitive PII is stored in the EU-based data privacy vault, and Skyflow responds with de-identified data.
The client-side code sends the account request, now with de-identified data, to the server.
The server processes the request, storing the data (now de-identified and tokenized) in cloud storage in the “Oregon, US” region.
At the end of the week, your company’s Redshift instance in Tokyo, Japan, loads the data (already de-identified and tokenized) from cloud storage to perform analytics.

Deploying multiple vaults situated in different regions streamlines the management of your sensitive data, ensuring compliance with data residency requirements across all your markets.

The data privacy vault architecture significantly simplifies the complexities associated with data residency and compliance. Furthermore, by exempting Redshift (or any warehouse) from the compliance responsibilities tied to data residency, global analytics operations continue seamlessly within a single warehouse instance.

Final Thoughts

Compliance regulations, with their stringent data residency stipulations, necessitate businesses to maintain rigorous standards for data localization, protection, privacy, and security. Adhering to these regulations is essential to mitigating the risks associated with breaches, penalties, and potential damage to reputation. However, enterprises operating in various global regions, serving diverse customer bases, are left to deal with the complex task of navigating multiple regulatory landscapes.

Using data privacy vaults as your core infrastructure for customer PII offers a streamlined solution to simplify global compliance, particularly concerning AWS services and cloud storage.

With a data privacy vault, organizations gain the ability to centralize the security of all sensitive data, effectively removing AWS and cloud storage from their compliance scope. By deploying data privacy vaults in various regions, companies can ensure that sensitive data storage and transmission align with the specific laws and regulations of each operational jurisdiction, thereby enhancing their overall compliance and security posture.

If you have thoughts on this or questions about this approach, please reach out to me on LinkedIn.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.

The post De-scoping Your AWS Services from Data Residency Requirements appeared first on Software Engineering Daily.

Hugging Face with Sayak Paul

5 december 2023 | 47 min

The Data Cloud’s Cheese and Diamond Problem

4 december 2023 | min

In any given week, if you search the news for “data breach”, you’ll see headlines like the ones below.

Companies like MGM and Caesars spend millions of dollars on firewalls, SIEMs, HSMs, and a whole smorgasbord of cybersecurity tools and yet, they can’t protect your social security number.

From hotels and casinos to some of the most innovative technology companies in the world, why is it that companies with seemingly endless financial and talent resources can’t get a handle on their data security challenges?

I believe this is due to a fundamental misunderstanding about the nature of data that started over 40 years ago.

Back in the 1980s, as computers found their way more and more into businesses, we lived in a disconnected world. To steal someone’s data, you had to physically steal the box the data lived on. As a consequence, we assumed that all data is created equal, that all data is simply ones and zeros, but this is wrong. All data isn’t created equal, some data is special, and needs to be treated that way.

In this blog post, I share my thoughts on what I refer to as the “Cheese and Diamond Problem” and how this has led to the data security challenges companies face today. I also offer explore an alternative approach, a new way of thinking, a privacy by engineering approach that helps us move towards a world where security is the default, and not bolted on.

The Cheese and Diamond Problem

Imagine that in my house I have cheese and I have diamonds. As a gracious host, I want guests of my home to be able to access my cheese. They should be able to freely go into the refrigerator and help themselves to some delicious cheese and perhaps a cracker.

However, I don’t want just anyone to touch my diamonds. Perhaps my diamonds even have sentimental value because it’s a diamond ring that’s been passed down through many generations in my family. Clearly the diamond is special.

Yet, if I store my diamonds in the refrigerator next to my cheese, it makes controlling access to the diamonds much more challenging. By co-locating these very different objects, my refrigerator alone isn’t enough to make sure my wife has access to the diamonds and cheese, but my guests only have access to my cheese.

The rules of engagement for something like diamonds are completely different than the rules of engagement for cheese. We all understand this distinction when it comes to physical objects.

This is exactly why my passport and my children’s birth certificates aren’t in the junk drawer in my kitchen with my batteries and my flashlights. If someone breaks into my home and steals my batteries, it’s not that big a deal, but if someone steals my daughter’s birth certificate, then I not only feel like I’ve failed as a parent, but the information on her birth certificate is also now compromised forever. I can’t simply replace her date of birth.

Despite all of us intuitively understanding that some physical objects are different, that they’re special, we somehow miss this point when we work with data. We don’t apply this thinking to Personally Identifiable Information (PII). We treat it like any other form of transactional or application data. We stuff it in a database, pass it around, make a million copies, and this leads to a whole host of problems.

The PII Replication Problem

Let’s consider a simple example.

In the diagram below, which represents an abstraction of a modern system, a phone number is being collected in the front end of the application, perhaps during account creation. That phone number ends up being passed downstream through each node and edge of the graph and at each node, we potentially end up with a copy of the phone number.

We store it in our database, in the warehouse, but we may also end up with a copy in our log files and the backups of all these systems. Instead of just having one copy of the phone number, we now have many copies and we need to protect all those locations and control access consistently wherever the data is stored.

Imagine that instead of having one copy of your passport that you keep in a secure location, you made 10,000 copies and then distributed them all over the world. Suddenly keeping your passport safe becomes a much harder problem in all 10,000 locations than if you have one copy secure in your home.

But this is exactly what we do with data.

We copy it everywhere and then attempt to lock down the hatches across all these systems and keep the policies and controls in sync about who can see what, when, and where. Additionally, because of the Cheese and Diamond Problem, we can’t adequately govern access to the data because the intermixing of our data conflates the rules of engagement about who has access. This quickly becomes an intractable problem because businesses don’t know what they’re storing or where it is, leading to the world we live in now where major corporations have data breaches on a regular basis.

Not All Data is Equal

Businesses are collecting and processing more data than ever. With the explosion of generative AI, as much as we are in an AI revolution, we are also in a data revolution. We can’t have powerful LLMs without access to massive data.

Companies leverage their data to drive business decisions, product direction, help serve customers better, and even create new types of consumer experiences. However, as discussed, not all data is created equal, some data, like PII, is special.

Over time, we’ve recognized that other forms of data like encryption keys, secrets, and identity are special and need to be treated that way. There was a time when we stored secrets in our application code or database. We eventually realized that was a bad idea and moved them into secret managers.

Approaches to managing different types of sensitive data.

Despite this progress, we are still left without an accepted standard for the storage and management of sensitive PII data. PII deserves the same type of special handling. You shouldn’t be contaminating your database with customer PII.

Luckily there’s a solution to this problem originally pioneered by companies like Netflix, Google, Apple, and Goldman Sachs and now touted by the IEEE as the future of privacy engineering, the PII Data Privacy Vault.

The PII Data Privacy Vault

A data privacy vault isolates, protects, and governs access to sensitive customer data (i.e. PII) while also keeping it usable. With a vault approach, you remove PII from your existing infrastructure, effectively de-scoping it from the responsibility of compliance and data security.

A vault is a first principles architectural approach to data privacy and security, facilitating workflows like:

PII storage and management for regulated industries
PCI storage and payment orchestration
Data residency compliance
Privacy-preserving analytics
Privacy-preserving AI

Let’s go back to our example from earlier where we were collecting a phone number from the front end of an application.

In the vault world, the phone number is sent directly to the vault from the front end. From a security perspective, we ideally want to de-identify sensitive data as early in the life cycle as possible. The real phone number will only exist within the vault, it acts as a single source of truth that’s isolated and protected outside of the existing systems.

Example of using a data privacy vault to de-scope an application.

The vault securely stores the phone number and generates a de-identified reference in the form of a token that gets passed back to the front end. The token has no mathematical connection to the original data, so it can’t be reverse engineered to reveal the original value.

This way, even if someone steals the data, as what happened with the Capital One data breach, the tokenized data carries no value. In fact, Capital One was fined only because they failed to tokenize all regulated data, some records were purely encrypted and those records were compromised.

Revealing Sensitive Data

While it’s great to securely store sensitive data, if we simply lock it up and throw away the key, it’s not super useful. We store all this customer PII so we can use it.

For example, we may need to reveal some of the data to a customer support agent, an IT administrator, a data analyst, or to the owner of the data. In this case, if we absolutely need to reveal some of the data, we want to re-identify it as late as possible, for example during render. We also want to limit what a user has access to based on the operations they need to perform with the data. While I might be able to see my full phone number, a customer support agent likely only needs the last four digits of my phone number and an analyst maybe only needs the area code for executing geo-based analytics.

The vault facilitates all of these use cases through a zero trust model where no one and no thing has access to data without explicit policies in place. The policies are built bottoms up, granting access to specific columns and rows of PII. This allows you to control who sees what, when, where, for how long, and in what format.

Let’s consider the situation where we have a user logging into an application and navigating to their account page. On the account page, we want to show the user their name, email, phone number, and home address based on the information they registered with us.

In the application database, we’ll have a table similar to the one shown below where the actual PII has been replaced by de-identified tokens.

Example of users table within the application database.

As in the non-vault world, the application will query the application database for the user record associated with the logged in user. The record will be passed to the front end application and the front end will exchange the tokens for a representation of the original values depending on the policies in place.

In the image below, the front end already has the tokenized data but needs to authenticate with the vault attaching the identity of the logged in user so that access is restricted based on the contextual information of the user’s identity. This is known as context-aware authorization.

Once authenticated and authorized, the front end can directly call the data privacy vault to reveal the true values of the user’s account information. But the front end only has access to this singular row of data and it’s limited to the few columns needed to render the information on the account page.

Example of revealing sensitive data for a single record.

Sharing Sensitive Data

No modern application exists in a silo. Most applications need to share customer PII with third party services to send emails, SMS, issue a payment, or some other type of workflow. This is also supported by the vault architecture by using the vault as a proxy to the third party service.

In this case, instead of calling a third party API directly, you call the data privacy vault with the de-identified data. The vault knows how to re-identify the PII securely within its environment, and then securely share that with the third party service.

An example of this flow for sending HIPAA compliant forms of communication is shown below. The backend server calls the vault directly with tokenized data and the vault then shares the actual sensitive data with the third party communication service.

Example of using a vault to send HIPAA compliant communication.

Final Thoughts

We’ve come a long way since building business applications in the 1980s, but we’ve failed to evolve our thinking regarding how we secure and manage customer PII. Point solutions like firewalls, encryption, and tokenization alone aren’t enough to address the fundamental problem. We need a new approach to cut to the root of the Cheese and Diamond Problem.

Not all data is the same, PII belongs in a data privacy vault.

The data privacy vault provides such an approach.

It’s an architectural approach to data privacy where security is the default. Multiple techniques like polymorphic encryption, confidential computing, tokenization, data governance, and others combine with the principle of isolation and zero trust to give you all the tools you need to store and use PII securely without exposing your systems to the underlying data.

If you have comments or questions about this approach, please connect with me on LinkedIn. Thanks for reading!

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.

The post The Data Cloud’s Cheese and Diamond Problem appeared first on Software Engineering Daily.

Building a Privacy-Preserving LLM-Based Chatbot

27 november 2023 | min

As Large Language Models (LLMs) and generative AI continue to grow more sophisticated and available, many organizations are starting to build, fine-tune, and customize LLMs based on their internal data and documents. This can bring incredible efficiency and reliability to data-driven decision-making processes. However, this practice comes with its share of challenges, primarily around data privacy, protection, and governance.

Let’s consider the construction of the LLM itself, which is trained on a massive amount of data collected from public and private sources. Without careful anonymization and filtering, sensitive data — such as PII or intellectual property — may be inadvertently included in the training set, potentially leading to a privacy breach.

Furthermore, privacy concerns are introduced when interacting with LLMs, as users might input sensitive data, such as names, addresses, or even confidential business information. If these inputs aren’t handled properly, the misuse or exposure of this information is a genuine risk.

In this post, we’ll explore how to work with LLMs in a privacy-preserving way when building an LLM-based chatbot. As we walk through the technology from end-to-end, we’ll highlight the most acute data privacy concerns and we’ll show how using a data privacy vault addresses those concerns.

Let’s start by taking a closer look at the problem we need to solve.

The problem: Protecting sensitive information from exposure by a chatbot

Consider a company that has uses an LLM-based chatbot for its internal operations. The LLM for the chatbot was built by modifying a pre-existing base model with embeddings created from internal company documents. The chatbot provides an easy-to-use interface that lets non-technical users within the company access information from internal data and documents.

The company has a sensitive internal project called “Project Titan.” Project Titan is so important and so sensitive that only people working on Project Titan know about it. In fact, the team often says: the first rule of Project Titan is don’t talk about Project Titan. Naturally, the team wants to take advantage of the internal chatbot and also include Project Titan specific information to speed up creation of design documents, documentation, and press releases. However, they need to control who can see details about this sensitive project.

What we have is a tangible and pressing privacy concern that sits at the intersection of AI and data. These challenges appear extremely difficult to solve in a scalable and production-ready way. Simply having a private version of the LLM doesn’t address the core issue of data access.

The proposed solution: Sensitive data de-identification and fine-grained access control

Ultimately, we need to identify the key points where sensitive data must be de-identified during the process of building (or fine-tuning) the LLM and the end user’s interaction with the LLM-based chatbot. After careful analysis, we’ve identified that there are two key points in the process where we need to de-identify (and later re-identify) sensitive data:

Before ingestion: When documents from Project Titan are used to create embeddings, the project name, any PII, and anything else sensitive to the project must be de-identified. This de-identification should occur as part of the ETL pipeline prior to data ingestion into the LLM.
During use: When a user inputs data to the chatbot, any sensitive data included in that input must also be de-identified.

You can de-identify sensitive data using Skyflow’s polymorphic encryption and tokenization engine that’s included within Skyflow Data Privacy Vault. This includes detection of PII but also terms you define within a sensitive data dictionary, like intellectual property (i.e. Project Titan).

Of course, only Project Titan team members who use the chatbot should be able to access the sensitive project data. Therefore, when the chatbot forms a response, we’ll rely on Skyflow’s governance engine (which provides fine-grained access control) and detokenization API to retrieve the sensitive data from the data privacy vault, making it available only to authorized end users.

Before we dive into the technical implementation, let’s go through a brief overview of foundational LLM concepts. If you’re already familiar with these concepts, you can skip the next section.

A brief primer on LLMs

LLMs are sophisticated artificial intelligence (AI) systems designed to analyze, generate, and work with human language. Built on advanced machine learning architectures, they are trained on vast quantities of text data, enabling them to generate text that is convincingly human-like in its coherence and relevance.

LLMs leverage a technology called transformers — one example is GPT, which stands for Generative Pre-Trained Transformer — to predict or generate a piece of text when given input or context. LLMs learn from patterns in the data they are trained on and then apply these learnings to understand newly given content or to generate new content.

Despite their benefits, LLMs pose potential challenges in terms of privacy, data security, and ethical considerations. This is because LLMs can inadvertently memorize sensitive information from their training data or generate inappropriate content if not properly regulated or supervised. Therefore, the use of LLMs necessitates effective strategies for data handling, governance, and preserving user privacy.

A technical overview of the solution

When embarking on any LLM project, we need to start with a model. Many open-source LLMs have been released in recent months, each with its specific area of focus. Instead of building an entire LLM model from scratch, many developers choose a pre-built model and then adjust the model with vector embeddings generated from domain-specific data.

Vector embeddings encapsulate the semantic relationship between words and help algorithms understand context. The embeddings act as an additional contextual knowledge base to help augment the facts known by the base model.

In our case, we’ll begin with an existing model from Hugging Face, and then customize it with embeddings. Hugging Face provides ML infrastructure services as well as open-source models and datasets.

In addition to the Hugging Face model, we’ll use the following additional tools to build out our privacy-preserving LLM-based ETL pipeline and chatbot:

LangChain an open-source Python library that chains together components typically used for building applications (such as chatbots) powered by LLMs
Snowflake, which we’ll use for internal document and data storage
Snowpipe, which we’ll use with Snowflake for automated data loading
Chroma, an AI-native, open-source database for vector embeddings
Streamlit, an open-source framework for building AI/ML-related applications using Python
RetrievalQA, a question-answering chain in LangChain which gets documents from a Retriever and then uses a QA chain to answer questions from those documents

The following diagram shows the high-level ETL and embeddings data flow:

Example of the ETL and embeddings data flow.

The ETL and embeddings flows from end to end are:

ETL

Start with source data, which may contain sensitive data.
Send data to Skyflow Data Privacy Vault for de-identification.
Use Snowpipe to load clean data into Snowflake.

Create vector embeddings

Load documents from Snowflake into LangChain.
Create vector embeddings with LangChain.
Store embeddings in Chroma.

Once the model has been customized with the Project Titan information, the user interaction and inference flow is as follows:

User interaction and inference information flow

Chat UI input

Accept user input via Streamlit’s chat UI.
Send user input to Skyflow for de-identification.

2. Retrieve embeddings

Get the embeddings from Chroma and attach to RetrievalQA.

3. Inference

Send clean data to RetrievalQA.
Use QA chain in RetrievalQA to answer the user’s question.

4. Chat UI response

Send RetrievalQA’s response to Skyflow for detokenization.
Send re-identified data to Streamlit for display to the end user.

Now that we’re clear on the high-level process, let’s dive in and take a closer look at each step.

ETL: Cleaning the source data

Cleaning the source data with Skyflow Data Privacy Vault is fairly straightforward and I’ve covered some of this in a prior post. In this case, we need to process all the source documents for Project Titan available in an AWS S3 bucket.

Skyflow will store the raw files, de-identify PII and IP, and save the clean files to another S3 bucket.

import boto3
from skyflow.vault import ConnectionConfig, Configuration, RequestMethod

# Authentication to Skyflow API
bearerToken = ''
def tokenProvider():
    global bearerToken
    if is_expired(bearerToken):
        return bearerToken
    bearerToken, _ = generate_bearer_token('<YOUR_CREDENTIALS_FILE_PATH>')
    return bearerToken

def processTrainingData(trainingData):
    try:
        # Vault connection configuration
        config = Configuration('<YOUR_VAULT_ID>', '<YOUR_VAULT_URL>', tokenProvider)

        # Define the connection API endpoint
        connectionConfig = ConnectionConfig('<YOUR_CONNECTION_URL>', RequestMethod.POST,
        requestHeader = {
            'Content-Type': 'application/json',
            'Authorization': '<YOUR_CONNECTION_BASIC_AUTH>'
        }
        requestBody = {
            'trainingData': trainingData
        }
 
        # Connect to the vault
        client = Client(config)
    
        # Call the Skyflow API to de-identify the training data
        response = client.invoke_connection(connectionConfig)

        # Define the S3 bucket name and key for the file
        bucketName = "clean-data-bucket"
        fileKey = "{timestamp}-{generated-uuid}"

        # Write the data to a file in memory
        fileContents = bytes(response.training_data.encode("UTF-8"))

        # Upload the file to S3
        s3 = boto3.client("s3")
        s3.put_object(Bucket=bucketName, Key=fileKey, Body=fileContents)
    except SkyflowError as e:
        print('Error Occurred:', e)

Next, we’ll configure Snowpipe to detect new documents in our S3 bucket and load that data into Snowflake. To do this, we’ll need to create the following in Snowflake:

CREATE OR REPLACE TABLE custom_training_data (
  training_text BINARY
  );

CREATE OR REPLACE FILE FORMAT training_data_json_format
  TYPE = JSON;

CREATE OR REPLACE TEMPORARY STAGE training_data_stage
 FILE_FORMAT = training_data_json_format;

CREATE PIPE custom_training_data
  AUTO_INGEST = TRUE
  AS
  COPY INTO custom_training_data
    FROM (SELECT $1:records.fields.training_text
          FROM @ training_data_stage t)
    ON_ERROR = 'continue';

With that, we have raw data that goes through a de-identification process, and then we store the plaintext sensitive data in Snowflake. Any sensitive data related to Project Titan is now obscured in the LLM, but because of Skyflow’s polymorphic encryption and tokenization, the de-identified data has referential integrity, meaning we can return the data to its original form when interacting with the chatbot.

Creating vector embeddings: Customizing our LLM

Now that we have our de-identified text data stored in Snowflake, we’re confident that all information related to Project Titan has been properly concealed. The next step is to create embeddings of these documents.

We’ll use the Instructor model provided by Hugging Face as our embedding model. We store our embeddings in Chroma, a vector database built expressly for this purpose. This will allow for the downstream retrieval and search support of the textual data stored in our vector database.

The code below loads the base model, embedding model, and storage context.

from langchain.chat_models import ChatOpenAI
from langchain.embeddings import HuggingFaceEmbeddings
from langchain.embeddings.openai import OpenAIEmbeddings

model_id = "hkunlp/instructor-large"
embed_model = HuggingFaceEmbeddings(model_name=model_id)
vectorstore = Chroma("langchain_store", embed_model)

Next, we need to load all documents and add them to the vector store. For this, we use the Snowflake document loader in LangChain.

from snowflakeLoader import SnowflakeLoader
import settings as s

QUERY = "select training_text as source from custom_training_data"
snowflake_loader = SnowflakeLoader(
    query=QUERY,
    user=s.SNOWFLAKE_USER,
    password=s.SNOWFLAKE_PASS,
    account=s.SNOWFLAKE_ACCOUNT,
    warehouse=s.SNOWFLAKE_WAREHOUSE,
    role=s.SNOWFLAKE_ROLE,
    database=s.SNOWFLAKE_DATABASE,
    schema=s.SNOWFLAKE_SCHEMA,
    metadata_columns=["source"],
)
training_documents = snowflake_loader.load()

vector_store.add_documents(training_documents)

With the training document and vector store created, we create the question-answering chain.

qa = RetrievalQA.from_chain_type(llm=ChatOpenAI(temperature=0.2,model_name='gpt-3.5-turbo'),
                                 chain_type="stuff", 
                                 retriever=vector_store.as_retriever())
result = qa.run("What is Project Titan?")

This question (“What is Project Titan?”) will fail because the model doesn’t actually know about Project Titan, it knows about a de-identified version of the string “Project Titan”.

To issue a query like this, the query needs to be first sent through Skyflow to de-identify the string and then the de-identified version is passed to the model. We’ll tackle this next as we start to put the pieces together for our chat UI.

Chat UI Input: Preserving privacy of user-supplied data

We’re ready to focus on the chatbot UI aspect of our project, dealing with accepting and processing user input as well as returning results with Project Titan data detokenized when needed.

For this portion of the project, we will use Streamlit for our UI. The code below creates a simple chatbot UI with Streamlit.

import openai
import streamlit as st

st.title("Acme Corp Assistant")

# Initialize the chat messages history
if "messages" not in st.session_state.keys():
    st.session_state.messages = [
        {"role": "assistant", "content": "Hello !  \nHow can I help?"}
    ]

# Prompt for user input and save
if prompt := st.chat_input():
    st.session_state.messages.append({"role": "user", "content": prompt})

# display the prior chat messages
for message in st.session_state.messages:
    with st.chat_message(message["role"]):
        st.write(message["content"])

# If last message is not from assistant, we need to generate a new response
if st.session_state.messages[-1]["role"] != "assistant":
    # Generate a response
    with st.chat_message("assistant"):
        with st.spinner("Thinking..."):
            response = "TODO"

    message = {"role": "assistant", "content": response}
    st.session_state.messages.append(message)

Our simple chat UI looks like this:

As you can see, the UI accepts a user input, but doesn’t currently integrate with our LLM. Next, we need to send the user input to Skyflow for de-identification before we use RetrievalQA to answer the user’s question. Let’s start with accepting and processing our input data.

To detect and de-identify plaintext sensitive data with Skyflow, we can use the detect API endpoint with code similar to the following:

def deIdentifyText(input):
   data = {
        "text": [
            {
                "message": input
            }
        ],
        "deidentify_option": "tokenize"
    }
    response = client.detect(data)

    return response[0].processed_text

Now that we’ve de-identified the user input data, we can send the question to RetrievalQA, which will then use a QA chain to answer the question from our documents.

def performCompletion(input):
     clean_input = deIdentifyText(input)

     qa = RetrievalQA.from_chain_type(llm=ChatOpenAI(temperature=0.2,model_name='gpt-3.5-turbo'),
                                 chain_type="stuff", 
                                 retriever=vector_store.as_retriever())
    return qa.run(clean_input)

We now have our response from RetrievalQA. However, we need to take one additional step before we can send it back to our user: detokenize (re-identify) our response through Skyflow’s detokenization API. This is fairly straightforward, similar to previous API calls to Skyflow.

Everything we need is encapsulated by the function performInference, which calls a function to reIdentifyText after the completion is returned.

Who can see what and in which format is controlled by Skyflow’s governance engine. There’s too much to cover here, but if you want to learn more, see Introducing the Skyflow Data Governance Engine.

def performInference(input):
    response = performCompletion(input)

    return reIdentifyText(response)

These final steps connect our entire application from end-to-end. Now, we need to update our UI code from above so that the response is correctly set.

# If last message is not from assistant, we need to generate a new response
if st.session_state.messages[-1]["role"] != "assistant":
    # Generate a response
    with st.chat_message("assistant"):
        with st.spinner("Thinking..."):
            response = performInference(m["content"])

With these pieces in place, here’s a quick demo of our privacy-preserving LLM-based chatbot in action:

*Example of the privacy-preserving bot in action.*

Tying it all together

In this article, we walked through the general steps to construct a privacy-preserving LLM-based chatbot. With organizations increasingly using LLM-based applications in their businesses and operations, the need to preserve data privacy has become acute. Concerns about protecting the privacy and security of sensitive data are the biggest adoption blocker that prevents many companies from making full use of AI with their datasets.

Solving this problem requires identifying the key points where sensitive data might enter your system and need to be de-identified. When working with LLMs, those points occur during model training — both when building an LLM or customizing one — and at the user input stage. You can use Skyflow Data Privacy Vault to implement effective de-identification and data governance for LLM-based AI tools like chatbots.

Building an LLM-based chatbot requires the use of several tools to ensure that data is handled in a manner that preserves privacy. Taking privacy-preserving measures is critical to prevent the misuse or exposure of sensitive information. By using the tools and methods we’ve demonstrated here, companies can leverage AI’s benefits and promote efficient data-driven decision-making while prioritizing data privacy and protection.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.

The post Building a Privacy-Preserving LLM-Based Chatbot appeared first on Software Engineering Daily.

Daytona with Ivan Burazin

23 november 2023 | 48 min

GraphAware with Luanne Misquitta

22 november 2023 | 58 min

One Snowflake, Multiple Vaults: A Solution to Data Residency

20 november 2023 | min

Data residency requirements, which govern where sensitive data can be stored or processed in the cloud (or in an on-prem server) are a common feature of many modern data protection laws. Because of data residency requirements, the location of sensitive data has significant regulatory compliance implications in countries and regions around the world.

In this post, we’ll look at the challenges of managing data residency with Snowflake. We’ll start by examining how Snowflake Cloud Regions address data residency challenges, and consider the compliance implications of this approach — especially when loading data from cloud storage. Then, we’ll look at how to simplify data residency compliance using one or more regional data privacy vaults.

Let’s begin with a deeper dive into data residency, and how it impacts compliance.

The implications of data residency on compliance

When you work with personally identifiable information (PII), where you store and process this information has a direct impact on your legal compliance requirements. Some jurisdictions have regulations that govern the protection and privacy of their residents’ PII, restricting how and where it’s used by businesses and other organizations.

For example, the personal data (i.e., PII) of European Union residents cannot be transferred outside the EU without appropriate safeguards.

The laws of each jurisdiction impact how you transmit, manage, process, and store sensitive data in that jurisdiction. Because data residency dictates where (geographically ) data is stored in the cloud, data residency becomes a critical concern in cloud environments that handle sensitive data.

Choose your cloud region carefully

Cloud service providers have data centers located in multiple regions around the world. When businesses sign up for cloud services and configure storage regions and other tooling, they select specific regions where their data is stored.

For many businesses, the selection of regions and locations for data storage is an afterthought.

But, treating this decision as an afterthought is a costly mistake that can come back to haunt you if you’re handling sensitive data. That’s because choosing storage regions is a weighty decision that can have a long-term impact on compliance, and on your business operations.

Snowflake Cloud Regions: a data residency solution?

Snowflake Cloud Regions let you choose the geographic location where your Snowflake data is stored across the data centers provided by the Snowflake-supported public cloud providers — AWS, GCP, and Azure. Each cloud provider offers a set of regions across the globe, with specific geographic data center locations in each cloud provider region.

*Source: Snowflake Documentation Supported Cloud Regions*

If your company uses Snowflake Cloud Regions, you have your choice of providers, as well as regions where your data can be stored. When you create an account to deploy and set up Snowflake, whichever region you select becomes the primary location for data storage and for data processing resources.

At first glance, it might seem like Snowflake Cloud Regions provides a simple, effective solution to your data residency and compliance concerns. But for global companies who need global analytics, it isn’t that simple. That’s because, as noted in the Snowflake Cloud Regions documentation:

Each Snowflake account is hosted in a single region. If you wish to use Snowflake across multiple regions, you must maintain a Snowflake account in each of the desired regions.

This means that for each region where your business operates that has data residency requirements, you’ll need a different Snowflake account hosted in that region. Compliance becomes increasingly complex as you scale globally to more and more regions around the world. With this approach, running global analytics operations across different accounts to get a comprehensive view of your business can be a massive and ongoing challenge.

Instead of managing multiple Snowflake accounts with multiple Snowflake instances distributed in various regions around the world, you’d rather maintain a Snowflake instance in a single region to support global data operations. However, you still need to consider the need to honor data residency requirements for sensitive data so you can uphold your compliance obligations and safeguard customer trust.

For example, if you collect the personal data (PII) of customers located in the EU, but your Snowflake instance is located somewhere else, then you need to think through the privacy and compliance impact of storing and processing that data.

Loading data from cloud storage into Snowflake

Snowflake also lets businesses load data from cloud storage services like AWS S3, Google Cloud Storage, Microsoft Azure — regardless of which cloud platform hosts the businesses’ Snowflake account. This can present additional challenges when working to ensure data residency compliance.

For example, let’s say that your company collects PII from both US and EU customers using its website. And, let’s say that this sensitive data is then stored in a Google Cloud Storage bucket that’s located in the AUSTRALIA-SOUTHEAST1 (Sydney) region.

How does transmitting this PII data to Australia, and then storing it in Australia, affect your compliance with regulations like the EU’s GDPR?

The answer is: doing this likely puts you out of compliance with GDPR. This is just one example of how the location where sensitive data is stored — and where it’s processed and replicated — complicates the compliance requirements faced by businesses that handle sensitive PII.

Businesses that handle PII must ensure regulatory compliance by aligning their choice of cloud storage regions with the data residency requirements of markets where they operate.

And beyond compliance issues, businesses should also consider data transfer costs. Transferring data between cloud storage regions can incur significant additional costs, especially if your company is frequently transferring large volumes of data. So, we not only have compliance concerns with cross-border transfers of PII, we also have a cost concern.

So, to briefly recap our problem:

Countries and regions have their own laws and regulations that govern how to handle their residents’ sensitive data (PII).
The geographic location where your business stores and processes sensitive data impacts whether you’re compliant with the data residency requirements of the markets where you operate.
If you use Snowflake to perform analytics on PII, then the complexity of meeting your compliance obligations will depend on the location of your Snowflake account.
If you load PII data into Snowflake from cloud storage, then your compliance obligations are also impacted by the location of your cloud storage.

So, how can we meet data residency requirements, support global analytics operations, and remove the operational overhead of managing multiple Snowflake accounts and instances?

We can solve our data residency problems and protect sensitive data with one or more data privacy vaults.

How a data privacy vault simplifies data privacy

A data privacy vault isolates, protects, and governs access to sensitive customer data. Sensitive data is stored in the vault, while opaque tokens that serve as references to this data are stored in traditional cloud storage or used in data warehouses. A data privacy vault can store sensitive data in a specific geographic location, and tightly controls access to this data. Other systems only have access to non-sensitive tokenized data.

In the example architecture shown below, a phone number is collected by a front end application. Ideally, we should de-identify (i.e., tokenize) this sensitive information as early in the data lifecycle as possible. A data privacy vault lets us do just that.

This phone number, along with any other PII, is stored securely in the vault, which is isolated outside of your company’s existing infrastructure. Any downstream services — the application database, data warehouse, analytics, any logs, etc. — store only a token representation of the data, and are removed from the scope of compliance:

*Example of reducing compliance scope with a data privacy vault*

Snowflake handles only de-identified data

Because no sensitive data is stored outside the data privacy vault, your compliance scope is restricted to just the vault. This removes the compliance burden from your Snowflake instance.

*Example pipeline where sensitive data is isolated and protected within a data privacy vault*

To satisfy data residency requirements, we can extend this approach by using multiple regional data privacy vaults placed near customers whose data is subject to these requirements. With sensitive data stored in these data privacy vaults, Snowflake contains only de-identified, tokenized data. It no longer matters if you operate a single global instance of Snowflake or multiple Snowflake accounts across different regions because data residency concerns no longer apply to your Snowflake instances.

Compliance with data residency requirements now depends solely on where your data privacy vaults are located. You no longer need to worry about data residency for all the different parts of your data tech stack, including cloud storage and Snowflake. All sensitive data goes into your data privacy vaults, and these vaults become the only component of your architecture subject to data residency requirements.

Store PII in a data privacy vault in a specific region

With Skyflow Data Privacy Vault you can host your vaults in a wide variety of regions around the world. You can also route sensitive data to a data privacy vault located in a specific region for storage.

For example, consider how the application architecture shown below supports data residency requirements from multiple regions:

*Using vaults to satisfy multiple data residency requirements for one Snowflake instance*

Your company’s e-commerce site collects customer PII whenever a customer places an order.
On the client side, the website detects the customers’ location.
Detecting that the customer is in the EU, the client-side code uses Skyflow’s API to send the PII data to your company’s data privacy vault in Frankfurt, Germany.
Note: For customers based in the US, the PII data is instead routed to the data privacy vault in the US (in this case, Virginia).
This EU-based customer’s sensitive PII is stored in the EU-based data privacy vault, and Skyflow’s API responds with tokenized data.
The client-side code sends the customer order request, now with tokenized data, to the server.
The server processes the order, storing the data (now de-identified and tokenized) in cloud storage in the “Oregon, US” region.
At the end of the week, your company’s Snowflake instance in Tokyo, Japan, loads the data (already de-identified and tokenized) from cloud storage to perform analytics.

By using multiple vaults located in different regions around the world, you can easily manage all of your sensitive data to meet various data residency compliance obligations across each of your global markets.

The data privacy vault architectural pattern vastly simplifies the challenges of data residency and compliance. Additionally, by de-scoping Snowflake from the compliance burden of data residency, global analytics executes as normal — within a single Snowflake instance.

Final thoughts

Compliance regulations and their data residency requirements require that businesses uphold stringent standards for data localization, protection, privacy, and security to reduce their risk of breaches, penalties, and reputational damage. However, businesses with customers (and data) located in a variety of global regions face the added challenge of managing multiple regulations across jurisdictions.

Using data privacy vaults lets businesses simplify their global compliance obligations around data residency as they relate to Snowflake and cloud storage.

Using a data privacy vault, companies can isolate and secure all sensitive data in one or more data privacy vaults, removing Snowflake and cloud storage from their compliance footprint. At the same time, by leveraging data privacy vaults in different regions, companies can help ensure that sensitive data is stored and transmitted according to the laws and regulations of each specific region where they operate.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.

The post One Snowflake, Multiple Vaults: A Solution to Data Residency appeared first on Software Engineering Daily.

Speechlab and Realtime Translation with Ivan Galea

2 november 2023 | 45 min

Sourcegraph with Quinn Slack

1 november 2023 | 42 min

If you’re a developer, there’s a good chance you’ve experimented with coding assistants like GitHub Copilot. Many developers have even fully integrated these tools into their workflows. One way these tools accelerate development is by autocompleting entire blocks of code. The AI achieves this by having awareness of the surrounding code. It understands context. However, in many cases the context available to an AI is limited. This restricts the AI’s ability to suggest more sweeping changes to a codebase, or even to refactor an entire application.

Quinn Slack is the CEO of Sourcegraph. He is now hard at work on the challenge of giving more context to AI – to make it aware of entire codebases, dependencies, error logs, and other data. Quinn joins the show today to talk about what it takes to move beyond code autocomplete, how to develop the next generation of coding AI, and what the future looks like for software engineers and programming languages.

Josh Goldberg is an independent full time open source developer in the TypeScript ecosystem. He works on projects that help developers write better TypeScript more easily, most notably on typescript-eslint: the tooling that enables ESLint and Prettier to run on TypeScript code. Josh regularly contributes to open source projects in the ecosystem such as ESLint and TypeScript.

Josh is a Microsoft MVP for developer technologies and the author of the acclaimed Learning TypeScript (O’Reilly), a cherished resource for any developer seeking to learn TypeScript without any prior experience outside of JavaScript. Josh regularly presents talks and workshops at bootcamps, conferences, and meetups to share knowledge on TypeScript, static analysis, open source, and general frontend and web development.

You can find Josh on: Bluesky, Fosstodon, Twitter, Twitch, YouTube, and joshuakgoldberg.com.

Please click here to view this show’s transcript.

Sponsorship inquiries: [email protected]

The post Sourcegraph with Quinn Slack appeared first on Software Engineering Daily.

Stack Overflow in the AI era with Ellen Brandenberger

3 oktober 2023 | 48 min

AI for Software Delivery with Birgitta Böckeler

15 augusti 2023 | 62 min

Ethical GPTs with Amruta Moktali

27 juli 2023 | 52 min

Data Investing and the MAD with Matt Turck

10 mars 2023 | 51 min

Surviving ChatGPT with Christian Hubicki

24 februari 2023 | 73 min

Edge Impulse with Daniel Situnayake

1 oktober 2022 | 49 min

Automatic Database Tuning with Andy Pavlo

23 september 2022 | 40 min

Practical Machine Learning in JavaScript with Charlie Gerard

18 september 2022 | 31 min

Union.ai with Ketan Umare

23 mars 2022 | 55 min

Ad-free Search on Neeva with Darin Fisher

25 januari 2022 | 49 min

Practical Machine Learning in JavaScript with Charlie Gerard

4 januari 2022 | 37 min

Responsibly Deploy AI in Production with Anupam Datta

30 november 2021 | 54 min

Learning Tensorflow.js with Gant Laborde

9 november 2021 | 51 min

No Code AI for Video Analytics with Alex Thiele

22 oktober 2021 | 43 min

Virtual Agents for IT and HR with Dan Turchin

22 september 2021 | 46 min

Autonomous Driving Infrastructure with Vinoj Kumar

20 september 2021 | 37 min

Sust Global: Taking Action Against Climate Change with Josh Gilbert

20 juli 2021 | 51 min

Machine Learning: The Great Stagnation with Mark Saroufim

4 juni 2021 | 51 min

Data Management Systems and Artificial Intelligence with Arun Kumar

27 maj 2021 | 63 min

BaseTen: Creating Machine Learning APIs with Tuhin Srivastava and Amir Haghighat

20 maj 2021 | 53 min

Botpress: Natural Language Processing with Sylvain Perron

7 maj 2021 | 50 min

MindsDB: Automated Machine Learning with Jorge Torres

6 april 2021 | 48 min

Creation Labs: Self Driving Trucks with Jakub Langr

30 mars 2021 | 49 min

Pinecone: Vector Database with Edo Liberty

15 mars 2021 | 46 min

OctoML: Automated Deep Learning Engineering with Jason Knight and Luis Ceze

9 februari 2021 | 54 min

iRobot with Chris Svec

29 januari 2021 | 57 min

Reinforcement Learning and Robotics with Nathan Lambert

27 januari 2021 | 55 min

Machine Learning Carbon Capture with Diego Saez-Gil

21 januari 2021 | 41 min

TensorFlow Lite with Pete Warden

11 januari 2021 | 41 min

WebAssembly on IoT with Jonathan Beri (Repeat)

5 januari 2021 | 50 min

Drishti: Deep Learning for Manufacturing with Krish Chaudhury (Repeat)

28 december 2020 | 53 min

Niantic Real World with Paul Franceus (Repeat)

22 december 2020 | 52 min

Practical AI with Chris Benson (Repeat)

17 december 2020 | 45 min

Kubeflow: TensorFlow on Kubernetes with David Aronchick (Repeat)

15 december 2020 | 56 min

Hedge Fund Artificial Intelligence with Xander Dunn (Repeat)

9 december 2020 | 53 min

Rosebud: Artificially Generated Media with Dzmitry Pletnikau

30 november 2020 | 48 min

Computer Architecture with Dave Patterson Holiday Repeat

27 november 2020 | 51 min

Cruise: Self-Driving Engineering with Mo Elshenawy Holiday Repeat

26 november 2020 | 49 min

Model Deployment and Serving with Chaoyu Yang

4 november 2020 | 38 min

Humanloop: NLP Model Engineering with Raza Habib

3 november 2020 | 45 min

Federated Learning with Mike Lee Williams

23 oktober 2020 | 52 min

Labelbox: Data Labeling Platform

19 oktober 2020 | 47 min

Roboflow: Computer Vision Models with Brad Dwyer

13 oktober 2020 | 49 min

Aquarium: Dataset Quality Improvement with Peter Gao

2 oktober 2020 | 58 min

Elementary Robotics with Arye Barnehama

17 september 2020 | 47 min

Robotic Process Automation with Antti Karjalainen

4 september 2020 | 43 min

Hyperparameter Tuning with Richard Liaw

28 augusti 2020 | 54 min

Machine Learning Labeling and Tooling with Lukas Biewald

26 augusti 2020 | 47 min

ParlAI: Facebook Dialogue Platform with Stephen Roller

20 augusti 2020 | 51 min

SuperAnnotate: Image Annotation Platform with Vahan and Tigran Petrosyan

19 augusti 2020 | 55 min

Drug Simulations with Bryan Vicknair and Jason Walsh

29 juli 2020 | 53 min

Metaflow: Netflix Machine Learning Platform with Savin Goyal

13 juli 2020 | 53 min

Determined AI: Machine Learning Ops with Neil Conway

8 juli 2020 | 42 min

Deepgram: End-to-End Speech Recognition with Scott Stephenson

3 juli 2020 | 45 min

Cresta: Speech ML for Calls with Zayd Enam

29 juni 2020 | 54 min

Traces: Video Recognition with Veronica Yurchuk and Kostyantyn Shysh (Summer Break Repeat)

25 juni 2020 | 62 min

Stripe Machine Learning Infrastructure with Rob Story and Kelley Rivoire (Summer Break Repeat)

16 juni 2020 | 65 min

Architects of Intelligence with Martin Ford (Summer Break Repeat)

15 juni 2020 | 59 min

Cruise Simulation with Tom Boyd

12 juni 2020 | 53 min

Tecton: Machine Learning Platform from Uber with Kevin Stumpf

3 juni 2020 | 53 min

Edge Machine Learning with Zach Shelby

26 maj 2020 | 57 min

Rasa: Conversational AI with Tom Bocklisch

24 april 2020 | 52 min

Snorkel: Training Dataset Management with Braden Hancock

9 april 2020 | 53 min

Descript with Andrew Mason

13 mars 2020 | 44 min

Anyscale with Ion Stoica

13 februari 2020 | 49 min

Machine learning applications are widely deployed across the software industry.

Most of these applications used supervised learning, a process in which labeled data sets are used to find correlations between the labels and the trends in that underlying data. But supervised learning is only one application of machine learning. Another broad set of machine learning methods is described by the term “reinforcement learning.”

Reinforcement learning involves an agent interacting with its environment. As the model interacts with the environment, it learns to make better decisions over time based on a reward function. Newer AI applications will need to operate in increasingly dynamic environments, and react to changes in those environments, which makes reinforcement learning a useful technique.

Reinforcement learning has several attributes that make it a distinctly different engineering problem than supervised learning. Reinforcement learning relies on simulation and distributed training to rapidly examine how different model parameters could affect the performance of a model in different scenarios.

Ray is an open source project for distributed applications. Although Ray was designed with reinforcement learning in mind, the potential use cases go beyond machine learning, and could be as influential and broadly applicable as distributed systems projects like Apache Spark or Kubernetes. Ray is a project from the Berkeley RISE Lab, the same place that gave rise to Spark, Mesos, and Alluxio.

The RISE Lab is led by Ion Stoica, a professor of computer science at Berkeley. He is also the co-founder of Anyscale, a company started to commercialize Ray by offering tools and services for enterprises looking to adopt Ray. Ion Stoica returns to the show to discuss reinforcement learning, distributed computing, and the Ray project.

If you enjoy the show, you can find all of our past episodes about machine learning, data, and the RISE Lab by going to SoftwareDaily.com and searching for the technologies or companies you are curious about . And if there is a subject that you want to hear covered, feel free to leave a comment on the episode, or send us a tweet @software_daily.

Sponsorship inquiries: [email protected]

The post Anyscale with Ion Stoica appeared first on Software Engineering Daily.

Practical AI with Chris Benson

9 december 2019 | 45 min

Future of Computing with John Hennessy Holiday Repeat

26 november 2019 | 57 min

Originally published June 7, 2018

Moore’s Law states that the number of transistors in a dense integrated circuit doubles about every two years. Moore’s Law is less like a “law” and more like an observation or a prediction.

Moore’s Law is ending. We can no longer fit an increasing amount of transistors in the same amount of space with a highly predictable rate. Dennard scaling is also coming to an end. Dennard scaling is the observation that as transistors get smaller, the power density stays constant.

These changes in hardware trends have downstream effects for software engineers. Most importantly–power consumption becomes much more important.

As a software engineer, how does power consumption affect you? It means that inefficient software will either run more slowly or cost more money relative to our expectations in the past. Whereas software engineers writing code 15 years ago could comfortably project that their code would get significantly cheaper to run over time due to hardware advances, the story is more complicated today.

Why is Moore’s Law ending? And what kinds of predictable advances in technology can we still expect?

John Hennessy is the chairman of Alphabet. In 2017, he won a Turing award (along with David Patterson) for his work on the RISC (Reduced Instruction Set Compiler) architecture. From 2000 to 2016, he was the president of Stanford University.

John joins the show to explore the future of computing. While we may not have the predictable benefits of Moore’s Law and Dennard scaling, we now have machine learning. It is hard to plot the advances of machine learning on any one chart (as we explored in a recent episode with OpenAI). But we can say empirically that machine learning is working quite well in production.

If machine learning offers us such strong advances in computing, how can we change our hardware design process to make machine learning more efficient?

As machine learning training workloads eat up more resources in a data center, engineers are developing domain specific chips which are optimized for those machine learning workloads. The Tensor Processing Unit (TPU) from Google is one such example. John mentioned that chips could become even more specialized within the domain of machine learning. You could imagine a chip that is specifically designed for a LSTM machine learning model.

There are other domains where we could see specialized chips–drones, self-driving cars, wearable computers. In this episode, John describes his perspective on the future of computing, and offers some framework for how engineers can adapt to that future.

Show Notes

The post Future of Computing with John Hennessy Holiday Repeat appeared first on Software Engineering Daily.

Incident Response Machine Learning with Chris Riley

12 november 2019 | 46 min

Traces: Video Recognition with Veronica Yurchuk and Kostyantyn Shysh

8 oktober 2019 | 61 min

Cruise: Self-Driving Engineering with Mo Elshenawy

1 oktober 2019 | 49 min

People.ai: Machine Learning for Sales with Andrey Akselrod

7 augusti 2019 | 45 min

A large sales organization has hundreds of sales people. Each of those sales people manages a set of accounts who they are trying to close sales deals on. Sales people are overseen by managers who ensure that the sales people are performing well. Directors and VPs ensure the scalability and health of the overall sales organization.

The sales lifecycle mostly takes place within a piece of software called a CRM: customer relationship management. This tool documents the interactions between sales people and accounts. CRMs have been around for many years, and although CRM software is a useful repository of data, it does not fulfill all the needs of a salesperson.

People.ai is a system of machine learning tools built around the sales tooling ecosystem. People.ai helps a sales organization avoid manual data entry, understand areas of potential improvement, and decide on who the highest value sales lead to pursue might be. Andrey Akselrod is the CTO At People.ai and he joins the show to discuss the potential applications of machine learning in the domain of sales, and the engineering work that his company has done.

Sponsorship inquiries: [email protected]

ANNOUNCEMENTS

FindCollabs is a place to find collaborators and build projects. We recently launched GitHub integrations. It’s easier than ever to find collaborators for your open source projects. And if you are looking for some people to start a project with, FindCollabs we have topic rooms that allow you to find other people who are interested in a particular technology, so that you can find people who are curious about React, or cryptocurrencies, or Kubernetes, or whatever you want to build with.
Podsheets is an open source podcast hosting platform that we recently launched. We are building Podsheets with the learnings from Software Engineering Daily, and our goal is to be the best place to host and monetize your podcast. If you have been thinking about starting a podcast, check out podsheets.com.
New SEDaily app for iOS and for Android. It includes all 1000 of our old episodes, as well as related links, greatest hits, and topics. You can comment on episodes and have discussions with other members of the community. I’ll be commenting on each episode, so if you hear an episode that you have some commentary on, jump onto the app, or on SoftwareDaily.com to share your thoughts. And you can become a paid subscriber for ad free episodes at softwareengineeringdaily.com/subscribe. Altalogy is the company who has been developing much of the software for the newest app, and if you are looking for a company to help you with your mobile and web development, I recommend checking them out.

The post People.ai: Machine Learning for Sales with Andrey Akselrod appeared first on Software Engineering Daily.

WebAssembly on IoT with Jonathan Beri

30 juli 2019 | 50 min

“Internet of Things” is a term used to describe the increasing connectivity and intelligence of physical objects within our lives.

IoT has manifested within enterprises under the term “Industrial IoT,” as wireless connectivity and machine learning have started to improve devices such as centrifuges, conveyor belts, and factory robotics. In the consumer space, IoT has moved slower than many people expected, and it remains to be seen when we will have widespread computation within consumer devices such as microwaves, washing machines, and lightswitches.

IoT computers have different constraints than general purpose computers. Security, reliability, battery life, power consumption, and cost structures are very different in IoT devices than in your laptop or smartphone. One technology that could solve some of the problems within IoT is WebAssembly, a newer binary instruction format for executable programs.

Jonathan Beri is a software engineer and the organizer of the San Francisco WebAssembly Meetup. He has significant experience in the IoT industry, and joins the show to discuss the state of WebAssembly, the surrounding technologies, and their impact on IoT.

ANNOUNCEMENTS

FindCollabs is a place to find collaborators and build projects. We recently launched GitHub integrations. It’s easier than ever to find collaborators for your open source projects. And if you are looking for some people to start a project with, FindCollabs we have topic rooms that allow you to find other people who are interested in a particular technology, so that you can find people who are curious about React, or cryptocurrencies, or Kubernetes, or whatever you want to build with.
Podsheets is an open source podcast hosting platform that we recently launched. We are building Podsheets with the learnings from Software Engineering Daily, and our goal is to be the best place to host and monetize your podcast. If you have been thinking about starting a podcast, check out podsheets.com.
New SEDaily app for iOS and for Android. It includes all 1000 of our old episodes, as well as related links, greatest hits, and topics. You can comment on episodes and have discussions with other members of the community. I’ll be commenting on each episode, so if you hear an episode that you have some commentary on, jump onto the app, or on SoftwareDaily.com to share your thoughts. And you can become a paid subscriber for ad free episodes at softwareengineeringdaily.com/subscribe. Altalogy is the company who has been developing much of the software for the newest app, and if you are looking for a company to help you with your mobile and web development, I recommend checking them out.

The post WebAssembly on IoT with Jonathan Beri appeared first on Software Engineering Daily.

Afresh: Grocery Store Software with Volodymyr Kuleshov

26 juni 2019 | 45 min

A grocery store contains fruit, vegetables, meat, bread, and other items that can expire. In order to keep these items in stock, the store must be aware of how much food has been sold and what has gone bad. When a food item is low in stock, the store needs to order more of that food from a central distribution system.

Managing food inventory is not simple. Some kinds of meat might expire faster than others. Avocados do not become ripe at the same rate as apples. In order to keep the shelves stocked, there are manual workflows for checking the inventory and ordering new inventory.

Afresh is a company that builds software for grocery stores.

Afresh works with grocery chains that have a central distribution center. These grocery stores already have some software. At the back of the store, inventory management systems maintain records of the items that the store has on the shelves. At the front of the store, checkout systems detect what has been sold and help to update inventory. When the inventory is running low, the store can order more inventory from the central distribution center, so that trucks can deliver more inventory.

Afresh improves the operational intelligence of these stores by detecting spoilage among items that are prone to expiration, such as fruit. Volodomyr Kuleshov is the CTO and co-founder of Afresh and he joins the show to discuss the technical challenges of a grocery store, and the software that Afresh is building to make groceries more intelligent.

ANNOUNCEMENTS

FindCollabs is a place to find collaborators and build projects. FindCollabs is the company I am building, and we are having an online hackathon with $2500 in prizes. If you are working on a project, or you are looking for other programmers to build a project or start a company with, check out FindCollabs. I’ve been interviewing people from some of these projects on the FindCollabs podcast, so if you want to learn more about the community you can hear that podcast.
New Software Daily app for iOS. It includes all 1000 of our old episodes, as well as related links, greatest hits, and topics. You can comment on episodes and have discussions with other members of the community. And you can become a paid subscriber for ad free episodes at softwareengineeringdaily.com/subscribe. Altalogy is the company who has been developing much of the software for the newest app, and if you are looking for a company to help you with your mobile and web development, I recommend checking them out.
Upcoming conferences I’m attending: Datadog Dash July 16th and 17th in NYC, Open Core Summit September 19th and 20th in San Francisco.
We are hiring two interns for software engineering and business development! If you are interested in either position, send an email with your resume to [email protected] with “Internship” in the subject line.

The post Afresh: Grocery Store Software with Volodymyr Kuleshov appeared first on Software Engineering Daily.

Niantic Real World with Paul Franceus

21 juni 2019 | 52 min

Niantic is the company behind Pokemon Go, an augmented reality game where users walk around in the real world and catch Pokemon which appear on their screen.

The idea for augmented reality has existed for a long time. But the technology to bring augmented reality to the mass market has appeared only recently. Improved mobile technology makes it possible for a smartphone to display rendered 3-D images over a video stream without running out of battery.

Ingress was the first game to come out of Niantic, followed by Pokemon Go, but there are other games on the way. Niantic is also working on the Niantic Real World platform, a “planet-scale” AR platform that will allow independent developers to build multiplayer augmented reality experiences that are as dynamic and entertaining as Pokemon Go.

Paul Franceus is an engineer at Niantic, and he joins the show to describe his experience building and launching Pokemon Go, as well as abstracting the technology from Pokemon Go and opening up the Niantic Real World platform to developers.

SHOW NOTES

Niantic developer contest finalists

ANNOUNCEMENTS

FindCollabs is a place to find collaborators and build projects. FindCollabs is the company I am building, and we are having an online hackathon with $2500 in prizes. If you are working on a project, or you are looking for other programmers to build a project or start a company with, check out FindCollabs. I’ve been interviewing people from some of these projects on the FindCollabs podcast, so if you want to learn more about the community you can hear that podcast.
New Software Daily app for iOS. It includes all 1000 of our old episodes, as well as related links, greatest hits, and topics. You can comment on episodes and have discussions with other members of the community. And you can become a paid subscriber for ad free episodes at softwareengineeringdaily.com/subscribe
Upcoming conferences I’m attending: Datadog Dash July 16th and 17th in NYC, Open Core Summit September 19th and 20th in San Francisco
We are hiring two interns for software engineering and business development! If you are interested in either position, send an email with your resume to [email protected] with “Internship” in the subject line.

The post Niantic Real World with Paul Franceus appeared first on Software Engineering Daily.

Stripe Machine Learning Infrastructure with Rob Story and Kelley Rivoire

13 juni 2019 | 63 min

Machine learning allows software to improve as that software consumes more data.

Machine learning is a tool that every software engineer wants to be able to use. Because machine learning is so broadly applicable, software companies want to make the tools more accessible to the developers across the organization.

There are many steps that an engineer must go through to use machine learning, and each additional step inhibits the chances that the engineer will actually get their model into production.

An engineer who wants to build machine learning into their application needs access to data sets. They need to join those data sets, and load them into a machine (or multiple machines) where their model can be trained. Once the model is trained, the model needs to test on additional data to ensure quality. If the initial model quality is insufficient, the engineer might need to tweak the training parameters.

Once a model is accurate enough, the engineer needs to deploy that model. After deployment, the model might need to be updated with new data later on. If the model is processing sensitive or financially relevant data, a provenance process might be necessary to allow for an audit trail of decisions that have been made by the model.

Rob Story and Kelley Rivoire are engineers working on machine learning infrastructure at Stripe. After recognizing the difficulties that engineers faced in creating and deploying machine learning models, Stripe engineers built out Railyard, an API for machine learning workloads within the company.

Rob and Kelley join the show to discuss data engineering and machine learning at Stripe, and their work on Railyard.

ANNOUNCEMENTS

FindCollabs is a place to find collaborators and build projects. FindCollabs is the company I am building, and we are having an online hackathon with $2500 in prizes. If you are working on a project, or you are looking for other programmers to build a project or start a company with, check out FindCollabs. I’ve been interviewing people from some of these projects on the FindCollabs podcast, so if you want to learn more about the community you can hear that podcast.
New Software Daily app for iOS. It includes all 1000 of our old episodes, as well as related links, greatest hits, and topics. You can comment on episodes and have discussions with other members of the community. And you can become a paid subscriber for ad free episodes at softwareengineeringdaily.com/subscribe
Upcoming conferences I’m attending: Datadog Dash July 16th and 17th in NYC, Open Core Summit September 19th and 20th in San Francisco
We are hiring two interns for software engineering and business development! If you are interested in either position, send an email with your resume to [email protected] with “Internship” in the subject line.

The post Stripe Machine Learning Infrastructure with Rob Story and Kelley Rivoire appeared first on Software Engineering Daily.

Augmented Reality Gaming with Tony Godar

28 maj 2019 | 45 min

Drishti: Deep Learning for Manufacturing with Krish Chaudhury

17 april 2019 | 52 min

Protein Structure Deep Learning with Mohammed Al Quraishi

15 april 2019 | 54 min

Machine Learning Joins with Arun Kumar

10 april 2019 | 59 min

Energy Market Machine Learning with Minh Dang and Corey Noone

11 mars 2019 | 43 min

Zoox Self-Driving with Ethan Dreyfuss

20 februari 2019 | 63 min

Store2Vec: DoorDash Recommendations with Mitchell Koch

19 februari 2019 | 50 min

Architects of Intelligence with Martin Ford

31 januari 2019 | 57 min

Artificial intelligence is reshaping every aspect of our lives, from transportation to agriculture to dating. Someday, we may even create a superintelligence–a computer system that is demonstrably smarter than humans. But there is widespread disagreement on how soon we could build a superintelligence. There is not even a broad consensus on how we can define the term “intelligence”.

Information technology is improving so rapidly we are losing the ability to forecast the near future. Even the most well-informed politicians and business people are constantly surprised by technological changes, and the downstream impact on society. Today, the most accurate guidance on the pace of technology comes from the scientists and the engineers who are building the tools of our future.

Martin Ford is a computer engineer and the author of Architects of Intelligence, a new book of interviews with the top researchers in artificial intelligence. His interviewees include Jeff Dean, Andrew Ng, Demis Hassabis, Ian Goodfellow, and Ray Kurzweil.

Architects of Intelligence is a privileged look at how AI is developing. Martin Ford surveys these different AI experts with similar questions. How will China’s adoption of AI differ from that of the US? What is the difference between the human brain and that of a computer? What are the low-hanging fruit applications of AI that we have yet to build?

Martin joins the show to talk about his new book. In our conversation, Martin synthesizes ideas from these different researchers, and describes the key areas of disagreement from across the field.

To find all 900 of our old episodes, including past episodes with authors and artificial intelligence researchers, check out the Software Engineering Daily app in the iOS and Android app stores. Whether or not you are a software engineer, we have lots of content about technology, business, and culture. In our app, you can also become a paid subscriber and get ad-free episodes–and you can have conversations with other members of the Software Engineering Daily community.

The post Architects of Intelligence with Martin Ford appeared first on Software Engineering Daily.

Kubeflow: TensorFlow on Kubernetes with David Aronchick

25 januari 2019 | 56 min

Human Sized Robots with Zach Allen

16 januari 2019 | 45 min

Word2Vec with Adrian Colyer Holiday Repeat

28 december 2018 | 55 min

Self-Driving Deep Learning with Lex Fridman Holiday Repeat

27 december 2018 | 52 min

Poker Artificial Intelligence with Noam Brown Holiday Repeat

21 november 2018 | 45 min

Reflow: Distributed Incremental Processing with Marius Eriksen

16 november 2018 | 65 min

The volume of data in the world is always increasing. The costs of storing that data is always decreasing. And the means for processing that data is always evolving.

Sensors, cameras, and other small computers gather large quantities of data from the physical world around us. User analytics tools gather information about how we are interacting with the Internet. Logging servers collect terabytes of records about how our systems are performing.

From the popularity of MapReduce, to the rise of open source distributed processing frameworks like Spark and Flink, to the wide variety of cloud services like BigQuery: there is an endless set of choices for how to analyze gigantic sets of data.

Machine learning training and inference is another dimension of the modern data engineering stack. Whereas tools like Spark and BigQuery are great for performing ad-hoc queries, systems like TensorFlow are optimized for the model training and deployment process.

Stitching together these tools allows a developer to compose workflows for how data pipelines progress through a data engineering system. One popular tool for this is Apache Airflow, which was created in 2014 and is widely used at companies like Airbnb.

Over the next few years, we will see a proliferation of new tools in the world of data engineering–and for good reason. There is a wealth of opportunity for companies to leverage their data to make better decisions, and potentially to clean and offer their internal data as APIs and pre-trained machine learning models.

Today, there is a vast number of enterprises who are modernizing their software development process with Kubernetes, cloud providers, and continuous delivery. Eventually, these enterprises will improve their complex software architecture, and will move from a defensive position to an offensive one. These enterprises will shift their modernization efforts from “DevOps” to “DataOps”, and thousands of software vendors will be ready to sell them new software for modernizing their data platform.

There is not a consensus for the best way to build and run a “data platform”. Nearly every company we have talked to on the show has a different definition and a different architecture for their “data platform”: Doordash, Dremio, Prisma, Uber, MapR, Snowflake, Confluent, Databricks…

We don’t expect to have a concise answer for how to run a data platform any time soon–but on the bright side, data infrastructure seems to be improving. Companies are increasingly able to ask questions about their data and get quick answers, in contrast to the data breadlines that were so prevalent five years ago.

Today we cover yet another approach to large scale data processing.

Reflow is a system for incremental data processing in the cloud. Reflow includes a functional, domain specific language for writing workflow programs, a runtime for evaluating those programs incrementally, and a scheduler for dynamically provisioning resources for those workflows. Reflow was created for large bioinformatics workloads, but should be broadly applicable to scientific and engineering computing workloads.

Reflow evaluates programs incrementally. Whenever the input data or the program changes, only the outputs that depend on the changes are recomputed. This minimizes the amount of recomputation that needs to be performed across a computational graph.

Marius Eriksen is the creator of Reflow and an engineer at GRAIL. He joins the show to discuss the motivation for a new data processing system–which involves explaining why workloads in bioinformatics are different than in some other domains.

The post Reflow: Distributed Incremental Processing with Marius Eriksen appeared first on Software Engineering Daily.

Computer Architecture with Dave Patterson

7 november 2018 | 51 min

Diffbot: Knowledge Graph API with Mike Tung

31 oktober 2018 | 50 min

Google Search allows humans to find and access information across the web. A human enters an unstructured query into the search box, the search engine provides several links as a result, and the human clicks on one of those links. That link brings up a web page, which is a set of unstructured data. Humans can read and understand news articles, videos, and Wikipedia pages.

Google Search solves the problem of organizing and distributing all of the unstructured data across the web, for humans to consume. Diffbot is a company with a goal of solving a related, but distinctly different problem: how to derive structure from the unstructured web, understand relationships within that structure, and allow machines to utilize those relationships through APIs.

Mike Tung is the founder of Diffbot. He joins the show to talk about the last decade that he has spent building artificial intelligence applications, from his research at Stanford to a mature, widely used product in Diffbot. I have built a few applications with Diffbot, and I encourage anyone who is a tinkerer or prototype builder to play around with it. It’s an API for accessing web pages as structured data.

Diffbot crawls the entire web, parsing websites, using NLP and NLU to comprehend those pages, and using probabilistic estimations to draw relationships between entities. It’s an ambitious product, and Mike has been working on it for a long time. I enjoyed our conversation.

Show Notes

We recently launched a new podcast: Fintech Daily! Fintech Daily is about payments, cryptocurrencies, trading, and the intersection between finance and technology. You can find it on fintechdaily.co or Apple and Google podcasts. We are looking for other hosts who want to participate. If you are interested in becoming a host, send us an email: [email protected]

The post Diffbot: Knowledge Graph API with Mike Tung appeared first on Software Engineering Daily.

Drift: Sales Bot Engineering with David Cancel

30 oktober 2018 | 54 min

David Cancel has started five companies, most recently Drift. Drift is a conversational marketing and sales platform. David has a depth of engineering skills and a breadth of business experience that make him an amazing source of knowledge. In today’s episode, David discusses topics ranging from the technical details of making a machine learning-driven sales platform to the battle scars from his early career, when he spent a lot of time building products that people did not want. He has found success by focusing on building software that the market has shown a desire for.

Chatbots were a popular, trendy subject a few years ago. The success of chatbots manifested in them fading into the background, and becoming a subtle, increasing part of our everyday interactions. Not every online interaction can be replaced by a chatbot, but many online interactions can be made more efficient by using chatbots. Chatbots can serve well-defined information, like product features, or the hours of operation of a business. When a chatbot gets a question that it cannot answer, the bot can route the conversation to a human.

When a customer lands on a web page of a company using Drift, they see a chat box appear in the corner of the screen. The customer is able to communicate through that chat box with a bot that represents the company. The customer can learn about the product, schedule a call with a salesperson, and get other useful utilities from the Drift sales bot.

The Drift chatbot messaging system is handled by Elixir/Erlang. Erlang is widely known as the messaging language that was used to scale WhatsApp while maintaining high availability. On the backend, Java services take the interactions from the Driftbot and pull it into a CRM, which allows sales and marketing people to manage information about the customers that are interacting with the chatbot. David gives lots more detail around the engineering stack, the deployment model, and his thoughts on the business and modern engineering.

We recently launched a new podcast: Fintech Daily! Fintech Daily is about payments, cryptocurrencies, trading, and the intersection between finance and technology. You can find it on fintechdaily.co or Apple and Google podcasts. We are looking for other hosts who want to participate. If you are interested in becoming a host, send us an email: [email protected]

The post Drift: Sales Bot Engineering with David Cancel appeared first on Software Engineering Daily.

Generative Models with Doug Eck

11 oktober 2018 | 61 min

Google Brain is an engineering team focused on deep learning research and applications. One growing area of interest within Google Brain is that of generative models. A generative model uses neural networks and a large data set to create new data similar to the ones that the network has seen before.

One approach to making use of generative models is GANs: generative adversarial networks. GANs can use a generative model (which creates new examples) together with a discriminator model (which can classify examples).

As an example, let’s take the task of generating new pictures of cats. We want an artificial cat picture generator. First, we train a discriminator by feeding it billions of example pictures of cats. We now have a model that can tell what a cat is. Next, we make a model that generates completely random images. We feed those randomly generated images to the discriminator. The discriminator outputs a “loss” for these random images. Loss is a metric we can use to represent how far off a given image is from being something that the discriminator would recognize as a cat. Finally, you can feed this “loss” back into the generative model, so that the generative model will adjust its weights in a way that will reduce loss. Over time, the generator gets better and better at reducing loss, until the discriminator starts believing that some of these semi-random images are actually cats.

Generative model systems have produced useful applications, such as object detection, image editing, and text-to-image generation. Today’s guest Doug Eck works on the Magenta team at Google Brain. Magenta uses applications of deep learning to produce tools and experiments around music, art, and creativity.

In a previous show, Doug described his vision for humans and computers to work together on creative tasks such as music. Today, we dive into some of the core machine learning building blocks that make machine creativity possible.

The post Generative Models with Doug Eck appeared first on Software Engineering Daily.

Real Estate Machine Learning with Or Hiltch

11 september 2018 | 51 min

RideOS: Fleet Management with Rohan Paranjpe

31 augusti 2018 | 58 min

Self-driving transportation will be widely deployed at some point in the future. How far off is that future? There are widely varying estimations: maybe you will summon a self-driving Uber in a New York within 5 years, or maybe it will take 20 years to work out all of the challenges in legal and engineering.

Between now and the self-driving future, there will be a long span of time where cars are semi-autonomous. Maybe your car is allowed to drive itself in certain areas of the city. Maybe your car can theoretically drive itself in 99% of conditions, but the law requires you to be behind the wheel until the algorithms get just a little bit better.

While we wait for self-driving to be widely deployed to consumers, a lot could change in the market. We know about Uber, Lyft, Waymo, Tesla and Cruise. But what about the classic car companies like Ford, Mercedes Benz, and Volkswagen? These companies are great at making cars, and they have hired teams of engineers working on self-driving.

But self-driving functionality is not the only piece of software you need to compete as a transportation company. You also need to build a marketplace for your autonomous vehicles, because in the future, far fewer people will want to own a car. Customers will want to use transportation as a service.

RideOS is a company that is building fleet management and navigation software. If you run a company that is building autonomous cars, you need to solve the problem of making an autonomous, safe robot that can drive you around.

Building an autonomous car is hard, but to go to market as a next-generation transportation company, you also need fleet management software, so you can deploy your cars in an Uber-like transportation system. And you need navigation software so that your cars know how to drive around.

RideOS lets a car company like Ford focus on building cars by providing a set of SDKs and cloud services for managing and navigating fleets of cars. Rohan Paranjpe joins today’s show to talk about the world of self-driving cars. Rohan worked at Tesla and Uber before joining RideOS, so he has a well-informed perspective on a few directions the self-driving car market might go in.

The post RideOS: Fleet Management with Rohan Paranjpe appeared first on Software Engineering Daily.

Stitch Fix Engineering with Cathy Polinsky

23 augusti 2018 | 51 min

DoorDash Engineering with Raghav Ramesh

16 augusti 2018 | 51 min

Self-Driving Engineering with George Hotz

8 augusti 2018 | 57 min

In the smartphone market there are two dominant operating systems: one closed source (iPhone) and one open source (Android). The market for self-driving cars could play out the same way, with a company like Tesla becoming the closed source iPhone of cars, and a company like Comma.ai developing the open source Android of self-driving cars.

George Hotz is the CEO of Comma.ai. Comma makes hardware devices that allow users with “normal” cars to be augmented with advanced cruise control and lane assist features. This means you can take your own car–for example, a Toyota Prius–and outfit your car to have something similar to the Tesla Autopilot. Comma’s hardware devices cost under $1000 to order online.

George joins the show to explain how the Comma hardware and software stack works in detail–from the low level interface with a car’s CAN bus to the high level machine learning infrastructure.

Users who purchase the Comma.ai hardware drive around with a camera facing the front of their windshield. This video is used to orient the state of the car in space. The video from that camera also gets saved and uploaded to Comma’s servers. Comma can use this video together with labeled events from the user’s driving experience to crowdsource their model for self-driving.

For example, if a user is driving down a long stretch of highway, and they turn on the Comma.ai driving assistance, the car will start driving itself and the video capture will begin. If the car begins to swerve into another lane, the user will take over for the car and the Comma system will disengage. This “disengagement” event gets labeled as such, and when that data makes it back to Comma’s servers, Comma can use the data to update their models.

George is very good at explaining complex engineering topics, and is also quite entertaining and open to discussing the technology as well as other competitors in the autonomous car space. I have not been able to get many other people on the show to talk about autonomous cars, so this was quite refreshing! I hope to do more in the future.

The post Self-Driving Engineering with George Hotz appeared first on Software Engineering Daily.

Botchain with Rob May

19 juli 2018 | 47 min

Machine Learning Deployments with Diego Oppenheimer

13 juli 2018 | 54 min

Machine learning models allow our applications to perform highly accurate inferences. A model can be used to classify a picture as a cat, or to predict what movie I might want to watch. But before a machine learning model can be used to make these inferences, the model must be trained and deployed.

In the training process, a machine learning model consumes a data set and learns from it. The training process can consume significant resources. After the training process is over, you have a trained model that you need to get into production. This is known as the “deployment” step.

Deployment can be a hard problem. You are taking a program from a training environment to a production environment. A lot can change between these two environments. In production, your model is running on a different machine–which can lead to compatibility issues. If your model serves a high volume of requests, it might need to scale up. In production, you also need caching, and monitoring, and logging.

Large companies like Netflix, Uber, and Facebook have built their own internal systems to control the pipeline of getting a model from training into production. Companies who are newer to machine learning can struggle with this deployment process, and these companies usually don’t have the resources to build their own machine learning platform like Netflix.

Diego Oppenheiner is the CEO of Algorithmia, a company that has built a system for automating machine learning deployments. This is the second cool product that Algorithmia has built, the first being the algorithm marketplace that we covered in an episode a few years ago.

In today’s show, Diego describes the challenges of deploying a machine learning model into production, and how that product was a natural complement to the algorithms marketplace. Full disclosure: Algorithmia is a sponsor of Software Engineering Daily.

The post Machine Learning Deployments with Diego Oppenheimer appeared first on Software Engineering Daily.

Machine Learning Stroke Identification with David Golan

5 juli 2018 | 57 min

Digital Evolution with Joel Lehman, Dusan Misevic, and Jeff Clune

15 juni 2018 | 51 min

Evolutionary algorithms can generate surprising, effective solutions to our problems.

Evolutionary algorithms are often let loose within a simulated environment. The algorithm is given a function to optimize for, and the engineers expect that algorithm to evolve a solution that optimizes for the objective function given the constraints of the simulated environment. But sometimes these results are not exactly what we are looking for.

For example, imagine an evolutionary algorithm that tries to evolve a creature that does a flip within a simulated physics engine that mirrors the real world.

You could imagine all sorts of evolutionary traits. Maybe the creature will evolve to have legs that are like springs, and let the creature jump high enough to do a flip. Maybe the creature will develop normal legs with strong muscles that propel the creature high enough to flip. But you wouldn’t expect the creature to evolve to be extremely tall–so tall that the creature can merely lean over fast enough so that the top of its body flips upside down. In one experiment, this is exactly what happened.

In another, similar experiment, the evolving creature discovered a bug in the physics engine of the simulated environment. This creature was able to exploit the problem with this physics engine to be able to move in ways that would not be possible in our real-world physical universe.

Evolutionary algorithms sometimes evolve solutions in ways that we don’t expect. Researchers usually throw those results away, because they don’t contribute to the result that the researchers are looking for. The consequence is that lots of interesting anecdotes get lost.

Joel Lehman, Dusan Misevic, and Jeff Clune are the lead authors of the paper “The Surprising Creativity of Digital Evolution.” The paper was a collection of anecdotes about strange results within the world of digital evolution. They join the show to describe what digital evolution is and some of the strange results that they surveyed in their paper.

Joel and Jeff are engineers at Uber’s artificial intelligence division–so this topic has applicable importance to them. Machine learning is all about evolution within simulated environments, and developing safe algorithms for AI requires an understanding of what can go wrong in a poorly defined evolutionary system.

The post Digital Evolution with Joel Lehman, Dusan Misevic, and Jeff Clune appeared first on Software Engineering Daily.

Future of Computing with John Hennessy

7 juni 2018 | 56 min

Moore’s Law states that the number of transistors in a dense integrated circuit double about every two years. Moore’s Law is less like a “law” and more like an observation or a prediction.

Moore’s Law is ending. We can no longer fit an increasing amount of transistors in the same amount of space with a highly predictable rate. Dennard scaling is also coming to an end. Dennard scaling is the observation that as transistors get smaller, the power density stays constant.

These changes in hardware trends have downstream effects for software engineers. Most importantly–power consumption becomes much more important.

As a software engineer, how does power consumption affect you? It means that inefficient software will either run more slowly or cost more money relative to our expectations in the past. Whereas software engineers writing code 15 years ago could comfortably project that their code would get significantly cheaper to run over time due to hardware advances, the story is more complicated today.

Why is Moore’s Law ending? And what kinds of predictable advances in technology can we still expect?

John Hennessy is the chairman of Alphabet. In 2017, he won a Turing award (along with David Patterson) for his work on the RISC (Reduced Instruction Set Compiler) architecture. From 2000 to 2016, he was the president of Stanford University.

John joins the show to explore the future of computing. While we may not have the predictable benefits of Moore’s Law and Dennard scaling, we now have machine learning. It is hard to plot the advances of machine learning on any one chart (as we explored in a recent episode with OpenAI). But we can say empirically that machine learning is working quite well in production.

If machine learning offers us such strong advances in computing, how can we change our hardware design process to make machine learning more efficient?

As machine learning training workloads eat up more resources in a data center, engineers are developing domain specific chips which are optimized for those machine learning workloads. The Tensor Processing Unit (TPU) from Google is one such example. John mentioned that chips could become even more specialized within the domain of machine learning. You could imagine a chip that is specifically designed for an LSTM machine learning model.

There are other domains where we could see specialized chips–drones, self-driving cars, wearable computers. In this episode, John describes his perspective on the future of computing and offers some framework for how engineers can adapt to that future.

Show Notes

The post Future of Computing with John Hennessy appeared first on Software Engineering Daily.

OpenAI: Compute and Safety with Dario Amodei

4 juni 2018 | 58 min

Applications of artificial intelligence are permeating our everyday lives. We notice it in small ways–improvements to speech recognition; better quality products being recommended to us; cheaper goods and services that have dropped in price because of more intelligent production.

But what can we quantitatively say about the rate at which artificial intelligence is improving? How fast are models advancing? Do the different fields in artificial intelligence all advance together, or are they improving separately from each other? In other words, if the accuracy of a speech recognition model doubles, does that mean that the accuracy of image recognition will double also?

It’s hard to know the answer to these questions.

Machine learning models trained today can consume 300,000 times the amount of compute that could be consumed in 2012. This does not necessarily mean that models are 300,000 times better–these training algorithms could just be less efficient than yesterday’s models, and therefore are consuming more compute.

We can observe from empirical data that models tend to get better with more data. Models also tend to get better with more compute. How much better do they get? That varies from application to application, from speech recognition to language translation. But models do seem to improve with more compute and more data.

Dario Amodei works at OpenAI, where he leads the AI safety team. In a post called “AI and Compute,” Dario observed that the consumption of machine learning training runs is increasing exponentially–doubling every 3.5 months. In this episode, Dario discusses the implications of increased consumption of compute in the training process.

Dario’s focus is AI safety. AI safety encompasses both the prevention of accidents and the prevention of deliberate malicious AI application.

Today, humans are dying in autonomous car crashes–this is an accident. The reward functions of social networks are being exploited by botnets and fake, salacious news–this is malicious. The dangers of AI are already affecting our lives on the axes of accidents and malice.

There will be more accidents, and more malicious applications–the question is what to do about it. What general strategies can be devised to improve AI safety? After Dario and I talk about the increased consumption of compute by training algorithms, we explore the implications of this increase for safety researchers.

The post OpenAI: Compute and Safety with Dario Amodei appeared first on Software Engineering Daily.

Voice with Rita Singh

21 maj 2018 | 57 min

Machine Learning with Data Skeptic and Second Spectrum at Telesign

19 maj 2018 | 70 min

Data Skeptic is a podcast about machine learning, data science, and how software affects our lives. The first guest on today’s episode is Kyle Polich, the host of Data Skeptic. Kyle is one of the best explainers of machine learning concepts I have met, and for this episode, he presented some material that is perfect for this audience: machine learning for software engineers.

Second Spectrum is a company that analyzes data from professional sports, turning that data into visualizations, reports, and futuristic sports viewing experiences. We had a previous show about Second Spectrum where we went into the company in detail–it was an excellent show, so I wanted to have Kevin Squire, an engineer from Spectrum, come on the show to talk about how the company builds machine learning tools to analyze sports data. If you have not seen any of the visualizations from Second Spectrum, stop what you are doing and watch a video on it!

This year we have had three Software Engineering Daily Meetups: in New York, Boston, and Los Angeles. At each of these Meetups, listeners from the SE Daily community got to meet each other and talk about software–what they are building and what they are excited about. I was happy to be in attendance at each of these, and I am posting the talks given by our presenters. The audio quality is not perfect on these, but there are also no ads.

Thanks to Telesign for graciously providing a space and some delicious food for our Meetup. Telesign has beautiful offices in Los Angeles, and they make SMS, voice, and data solutions. If you are looking for secure and reliable communications APIs, check them out.

We’d love to have you as part of our community. We will have more Meetups eventually, and you can be notified of these by signing up for our newsletter. Come to SoftwareDaily.com and get involved with the discussion of episodes and software projects. You can also check out our open source projects–the mobile apps, and our website.

The post Machine Learning with Data Skeptic and Second Spectrum at Telesign appeared first on Software Engineering Daily.

Deep Learning Topologies with Yinyin Liu

10 maj 2018 | 54 min

Algorithms for building neural networks have existed for decades. For a long time, neural networks were not widely used. Recent changes to the cost of compute and the size of our data have made neural networks extremely useful. Our smartphones generate terabytes of useful data. Lower storage costs make it economical to keep that data. Cloud computing democratized the ability to do large scale machine learning across deep learning hardware.

Over the last few years, these trends have been driving widespread use of deep learning, in which neural nets with a large series of layers are used to create powerful results in various fields of classification and prediction. Neural networks are a tool for making sense of unstructured data–text, images, sound waves, and videos.

“Unstructured” data is data with high volume or high dimensionality. For example, an image has a huge collection of pixels, and each pixel has a color value. One way to think about image classification is that you are finding correlations between those pixels. A certain cluster of pixels might represent an edge. After doing edge detection on pixels, you have a collection of edges. Then you can find correlations between those edges, and build up higher levels of abstraction.

Yinyin Liu is a principal engineer and head of data science at the Intel AI products group. She studies techniques for building neural networks. Each different configuration of a neural network for a given problem is called a “topology.” Engineers are always looking at new topologies for solving a deep learning application–such as natural language processing.

In this episode, Yinyin describes what a deep learning topology is and describes topologies for natural language processing. We also talk about the opportunities and the bottlenecks in deep learning–including why the tools are so immature, and what it will take to make the tooling better.

The post Deep Learning Topologies with Yinyin Liu appeared first on Software Engineering Daily.

Keybase Architecture / Clarifai Infrastructure Meetup Talks

28 april 2018 | 73 min

TensorFlow Applications with Rajat Monga

26 april 2018 | 51 min

Scale Self-Driving with Alexandr Wang

27 februari 2018 | 43 min

The easiest way to train a computer to recognize a picture of a cat is to show the computer a million labeled images of cats. The easiest way to train a computer to recognize a stop sign is to show the computer a million labeled stop signs.

Supervised machine learning systems require labeled data. Today, most of that labeling needs to be done by humans. When a large tech company decides to “build a machine learning model,” that often requires a massive amount of effort to get labeled data.

Hundreds of thousands of knowledge workers around the world earn their income from labeling tasks. An example task might be “label all of the pedestrians in this intersection.” You receive a picture of a crowded intersection, and your task is to circle all the pedestrians. You have now created a piece of labeled data.

Scale API is a company that turns API requests into human tasks. Their most recent release is an API for labeling data that has been generated from sensors. As self-driving cars emerge onto our streets, the sensors on these cars generate LIDAR, radar, and camera data. The cars will interpret that data in real-time using their machine learning models, and then they will send that data to the cloud so that the data can be processed offline to improve the machine learning models of every car on the road.

The first step in that processing pipeline is the labeling–which is the focus of today’s conversation. Alexandr Wang is the CEO of Scale, and he joins the show to discuss self-driving cars, labeling, and the company he co-founded.

A few notes before we get started. We just launched the Software Daily, job board. To check it out, go to softwaredaily.com/jobs. You can post jobs, you can apply for jobs, and it’s all free. If you are looking to hire, or looking for a job, I recommend checking it out. And if you are looking for an internship, you can use the job board to apply for an internship at Software Engineering Daily.

Also, Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA.

The post Scale Self-Driving with Alexandr Wang appeared first on Software Engineering Daily.

Machine Learning Deployments with Kinnary Jangla

14 februari 2018 | 41 min

Deep Learning Hardware with Xin Wang

29 januari 2018 | 52 min

Edge Deep Learning with Aran Khanna

26 januari 2018 | 51 min

A modern farm has hundreds of sensors to monitor the soil health, and robotic machinery to reap the vegetables. A modern shipping yard has hundreds of computers working together to orchestrate and analyze the freight that is coming in from overseas. A modern factory has temperature gauges and smart security cameras to ensure workplace safety.

All of these devices could be considered “edge” devices.

Over the last decade, these edge devices have mostly been used to gather data and save it to an on-premise server, or to the cloud. Today, as the required volumes of data and compute scale, we look for ways to better utilize our resources. We can start to deploy more application logic to these edge devices, and build a more sophisticated relationship between our powerful cloud servers and the less powerful edge devices.

The soil sensors at the farm are recording long time series of chemical levels. The pressure sensors in a centrifuge are recording months and years of data. The cameras are recording terabytes of video. These huge data sets are correlated with labeled events–such as crop yields.

With these large volumes of data, we can construct models for responding to future events. Deep learning can be used to improve systems over time. The models can be trained in the cloud and deployed to devices at the edge.

Aran Khanna is an AI engineer with Amazon Web Services, and he joins the show to discuss workloads at the cloud and at the edge–how work can be distributed between the two places, and the tools that can be used to build edge deep learning systems more easily.

To find all of our shows about machine learning and edge computing, as well as links to learn more about the topics described in the show, download the Software Engineering Daily app for iOS or Android. These apps have all 650 of our episodes in a searchable format–we have recommendations, categories, related links, and discussions around the episodes. It’s all free and also open source–if you are interested in getting involved in our open source community, we have lots of people working on the project and we do our best to be friendly and inviting to new people coming in looking for their first open-source project. You can find that project at Github.com/softwareengineeringdaily

The post Edge Deep Learning with Aran Khanna appeared first on Software Engineering Daily.

Machine Learning and Technical Debt with D. Sculley Holiday Repeat

25 december 2017 | 32 min

Training the Machines with Russell Smith

17 november 2017 | 60 min

Automation is changing the labor market.

To automate a task, someone needs to put in the work to describe the task correctly to a computer. For some tasks, the reward for automating a task is tremendous–for example, putting together mobile phones. In China, companies like FOXCONN are investing time and money into programming the instructions for how to assemble your phone. Robots execute those instructions.

FOXCONN spends millions of dollars deploying these robots, but it is a worthwhile expense. Once FOXCONN pays off the capital investment in those robots, they have a tireless workforce that can build phones all day long. Humans require training, rest, and psychological considerations. And with robots, the error rate is lower. Your smart phone runs your life, and you do not want the liability of human imperfection involved in constructing that phone.

As we race towards an automated future, the manual tasks that get automated first depend on their economic value. The manual labor costs of smartphone construction is a massive expense for corporations. This is also true for truck driving, food service, and package delivery. The savings that will be reaped from automating these tasks are tremendous–regardless of how we automate them.

There two ways of building automated systems: rule-based systems and machine learning.

With rule-based systems, we can describe to the computer exactly what we want it to do–like following a recipe. With machine learning, we can train the computer by giving it examples and let the computer derive its own understanding of how to automate a task.

Both approaches to automation have difficulties. A rule-based approach requires us to enumerate every single detail to the machine. This might work well in a highly controlled environment like a manufacturing facility. But rule-based systems don’t work well in the real world, where there are so many unexpected events, like snowstorms.

As we reported in a previous episode about how to build self-driving cars, engineers still don’t quite know what the right mix of rule-based systems and machine learning techniques are for autonomous vehicles. But we will continue to pour money into solving this problem, because the investment is worth figuring out how to train the machine.

The routine tasks in our world will be automated given enough time. How soon something will be automated depends on how expensive that task is when it is performed by a human, and how hard it is to design an artificial narrow intelligence to perform the task instead of a human.

Manual software testing is another type of work that is being automated today.

If I am building a mobile app to play podcast episodes, and I make a change to the user interface, I want to have manual quality assurance (QA) testers run through tests that I describe to them, to make sure my change did not break anything. QA tests describe high level application functionality. Can the user register and log in? Can the user press the play button and listen to a podcast episode on my app?

Unit tests are not good enough, because unit tests only verify the logic and the application state from the point of view of the computer itself. Manual QA tests ensure that the quality of the user experience was not impacted.

With so many different device types, operating systems, and browsers, I need my QA test to be executed in all of the different target QA environments. This requires lots of manual testers. If I want manual testing for every deployment I push, that manual testing can get expensive.

RainforestQA is a platform for QA testing that turns manual testing into automated testing. The manual test procedures are recorded, processed by computer vision, and turned into automated tests. RainforestQA hires human workers from Amazon Mechanical Turk to execute the well-defined manual tests, and the recorded manual procedure is used to train the machines that can execute the same task in the future.

Russell Smith is the CTO and co-founder of RainforestQA, and he joins the show to explain how RainforestQA works: the engineering infrastructure, the process of recruiting workers from mechanical turk, and the machine learning system for taking manual tasks and automating them.

Show Notes: Andrej Karpathy Turk Story

The post Training the Machines with Russell Smith appeared first on Software Engineering Daily.

Model Training with Yufeng Guo

18 oktober 2017 | 41 min

Sports Deep Learning with Yu-Han Chang and Jeff Su

29 september 2017 | 50 min

Deep Learning Systems with Milena Marinova

19 september 2017 | 48 min

Visual Search with Neel Vadoothker

15 september 2017 | 54 min

Word2Vec with Adrian Colyer

13 september 2017 | 55 min

Machines understand the world through mathematical representations. In order to train a machine learning model, we need to describe everything in terms of numbers. Images, words, and sounds are too abstract for a computer. But a series of numbers is a representation that we can all agree on, whether we are a computer or a human.

In recent shows, we have explored how to train machine learning models to understand images and video. Today, we explore words. You might be thinking–”isn’t a word easy to understand? Can’t you just take the dictionary definition?” A dictionary definition does not capture the richness of a word. Dictionaries do not give you a way to measure similarity between one word and all other words in a given language.

Word2vec is a system for defining words in terms of the words that appear close to that word. For example, the sentence “Howard is sitting in a Starbucks cafe drinking a cup of coffee” gives an obvious indication that the words “cafe,” “cup,” and “coffee” are all related. With enough sentences like that, we can start to understand the entire language.

Adrian Colyer is a venture capitalist with Accel, and blogs about technical topics such as word2vec. We talked about word2vec specifically, and the deep learning space more generally. We also explored how the rapidly improving tools around deep learning are changing the venture investment landscape.

If you like this episode, we have done many other shows about machine learning with guests like Matt Zeiler, the founder of Clarif.ai and Francois Chollet, the creator of Keras. You can check out our back catalog by downloading the Software Engineering Daily app for iOS, where you can listen to all of our old episodes, and easily discover new topics that might interest you. You can upvote the episodes you like and get recommendations based on your listening history. With 600 episodes, it is hard to find the episodes that appeal to you, and we hope the app helps with that.

Question of the Week: What is your favorite continuous delivery or continuous integration tool? Email [email protected] and a winner will be chosen at random to receive a Software Engineering Daily hoodie.

The post Word2Vec with Adrian Colyer appeared first on Software Engineering Daily.