It’s Not Rocket Science

Edwin Chiu - Expect Failure

"It's very rare that a client will hold and cradle the electronics box you've designed and hug it.”

Sources

Episode 4 — Edwin Chiu

From engine controllers to underwater robots to violin solos at Starship launches — Edwin Chiu on why making cappuccino can be more fulfilling than designing avionics for rockets.

Hosted by Jeff Ward · Former VP of Avionics & Software, SpaceX

Transcript has been lightly edited for readability. Timestamps reflect the published episode.

Jeff Ward   [0:00]

Hey friends, welcome to episode four of It's Not Rocket Science. My name is Jeff Ward, and for a few years I was the vice president of avionics at SpaceX, and I've been interviewing people about their SpaceX journeys. Super Heavy and Starship are on the pad again, and I thought it was time to edit the interview I did with avionics engineer Edwin Chiu. During the course of the discussion, he'll tell us why making cappuccino is more fulfilling than making engine controllers, why you might want a personal submarine, and what happens when Elon Musk asks you an interview riddle. Engine sequence initiated. 3, 2, 1.

Edwin Chiu   [0:33]

Do all the engineering you want, but it's very rare that a client will, you know, hold and cradle this electronics box that you've designed and hug it and just be like, ah, this is exactly what I needed. We have liftoff.

Jeff Ward   [0:45]

We have liftoff. There's a video that circulated on social media just after the first launch of Super Heavy and Starship. It shows a group of people on the beach watching the distant launch site, and just about the time of liftoff, the guy in the foreground turns to you, and you see that he's got a violin. And mixed in with the cheering of the crowd and the rumble of the Super Heavy, you can hear that he's playing Ring of Fire. This is my October 2021 interview with that violin player.

Edwin Chiu   [1:24]

My name is Edwin. I am a self-styled underwater circuits performer. My passions are in electrical engineering and exploration. So a lot of the things I think we'll talk about today are related to being underwater, exploring new things, and coffee.

Jeff Ward   [1:41]

I reviewed huge piles of resumes when I was at SpaceX, and the ones that made it out of the pile always had a hook. Edwin's hook was a bike.

Edwin Chiu   [1:49]

Yeah, the bike. So this bike started off as like a kid's project in fifth grade, but ended up getting me into a lot of places that perhaps I otherwise shouldn't have been. But it got me into all kinds of mostly good trouble. When I was a kid, there was an electric vehicle building club in my middle school. I was too young to drive, and also at fifth grade, too young to join the electric vehicle building club. So I talked to the teacher running that program, Mr. Brad Booth, who very kindly said, well, you know, rules are rules. I can't let you join the club, but I can help you do it on your own. And so at 10 years old, in fifth grade, I built my first electric scooter. I do believe I stole my sister's kick scooter and made it electric, which I don't know if you've forgiven me for that.

Jeff Ward   [2:34]

I think it's awesome because I often hold it up as an example of the kind of thing, that should go on a resume for an engineer who wants to get a cool job.

Edwin Chiu   [2:41]

So, you know, that's a funny story that when I was applying to college, my scores and grades weren't really high enough to kind of hit some of the reach schools that I had wanted to go to. But even though it wasn't allowed to add any supplemental documents to your application, at that time, all applications were mailed, right? So you could just slip something into the envelope. And my parents helped me make a brochure of my electric vehicle adventures in my younger years and just slipped it in each of those envelopes. And I got more responses than I expected from colleges.

 

Jeff Ward  

When you applied at SpaceX, you had intended to work there for a year?

 

Yeah. I think when I applied, the recruiter was very honest with me. They said, look, this is a startup. This is what we're trying to do is very hard and very new. And we have about six months of money left at the time of hiring you. So unless we get this thing off the ground, and of course, we think we can, but know that there's a very good chance that you may be looking for a job again in half a year. And also at that point, I had intended to take a break from school, go work in engineering in industry for a year, and then return and complete my PhD.

 

What I had realized was all the teachers that I respected the most had spent some time in industry before coming back to academia, and I wanted to do the same. So I thought I'd take a year out and the rest is history. I actually never went back.

 

Jeff Ward  

And what did you do at SpaceX?

 

Edwin Chiu  

My specialty is a mixed signal electrical controller. So the boxes that sit next to the engines, whether it's an engine in your car or an engine in a rocket, it's the computer that controls the firing and operation and data from the engine or various other devices around the rocket or spacecraft or a submarine. Worked on Grasshopper and then on Dragon doing sensors for the Dragon system. And then was part of the team that I started with Falcon 9 doing the engine controller, did the first architectural design for the electrical system of Dragon.

 

Jeff Ward  

Do you have a SpaceX story that you tell people, like the quintessential SpaceX story of your experience there?

 

Well, there are many, and there are many things that I learned from SpaceX, but I think the one that stands out most to me was I was at a barbecue drilling with the avionics folks, and I get a call from Katrina Chambers leading up the electronics saying, no time to talk, go home, get a bag, get on a plane. And, you know, you receive a phone call like that. You don't really ask too many questions. So I was on a plane to Texas and on the way, Katrina explained to me that we had found a water ingress issue in the design of the engine controller. And because the design of the engine controller also forms the basis for most of the other electronics boxes on the vehicle, they're all sort of built in the same way. Everything was at risk. We had looked at the data and said, you know, there might be a little moisture in the box. Open it up and see what's going on. And they sent back a video where they poured like a quart of water out of the engine controller. And this was, I think, five days away from a major integrated static fire test for Falcon 9. And so over the next five days, I think we flew back and forth to Texas two or three times. We hardly slept at all on the plane. We weren't sleeping. We were reviewing data. In those next five days, we took all of the electronics off of the rocket, brought them back to Hawthorne, either tested, repaired, or fixed all of them, and brought them back to McGregor in our carry-on luggage. And within a day, had reinstalled everything back on the rocket. And the static fire took place on the day that it was planned, not a day of schedule slip. And I think we worked something like 125 hours and five days.

Jeff Ward   [6:26]

Edwin did instantly reality check his own SpaceX hyperbole.

Edwin Chiu   [6:29]

Or no, 115. It was 115 hours and five days, which is close to the number of hours in five days. Yeah. And multiple flights across the country as well. Yeah. And that was the first flight and first successful flight of Falcon 9. What were your personal expectations for that first launch? Well, as you know, that's not common in the space business, that it usually takes three or four flights to get flying. And I think we knew this for Falcon 9. We had learned a lot of lessons with Falcon 1, but Falcon 9 was a completely different piece. Every step of the way, every 10 seconds, you had this feeling, well, if it only went this far, that wouldn't be enough. You know, 10 seconds, 20 seconds, 30 seconds, it kept flying. And stage separation, second stage ignition. And we're like, come on, come on, make it. Just a few more minutes. Daddy needs a new pair of shoes. Exactly. Right. Second stage orbit confirmation. All of these things, every step of the way was euphoric. It was absolutely wild. And that's the thing about the startup world too, is that, you know, you're not a big player, right? You only get so many chances. And if it takes four tries to get to orbit and you only have money for three, you never make it. And that was close at SpaceX with Falcon 1.

Jeff Ward   [7:40]

It took four times and we had money for four times.

Edwin Chiu   [7:43]

I believe what Elon told us is that we had money for three and we really scraped it for four. This is unconfirmed, but I do believe there were some parts from Lowe's hardware next door on that rocket. Yeah, it wouldn't surprise me at all. Did you have any encounters with Elon during those years at SpaceX that stick in your mind? Certainly. Like, you know, you usually don't interact with Elon unless something has gone very wrong. And I did interact with him a few times in the design and development of the Falcon 9 engine controller. But my first and probably most salient encounter with Elon was during my interview. I didn't realize it at the time, but Elon interviewed every single person who hired full-time at that time. And at the end of the interview, I made the mistake of asking him, hey, do you have any other questions for me? And Elon thought about it for a moment, was about to let me go and said, actually, I'll tell you a riddle. And he told me this riddle. And I was kind of coached by the recruiter not to allow this to happen. And as I did it, I realized that I had made the crucial mistake. And then Elon just spent the next half an hour and just stared at me as I sweat, soaked down to my socks, trying to solve this puzzle that he had given me. And, you know, he would check his email or, like, sort some paper on his desk. And he would just go right back to this. And I must have lost, like, a liter of water during that interview.

Jeff Ward   [9:04]

I was talking to Dolly Singh, who was a recruiter at that time. And she said that she was specifically looking for people, you know, as she put it, who wouldn't wet the bed.

Edwin Chiu   [9:14]

Yeah, well, I was pretty close, let me tell you.

 

Jeff Ward  

How did you end up moving on from SpaceX? It sounds like you stayed there a lot longer than you expected to.

Edwin Chiu

You know, SpaceX is not a place, I think, for many people to work a long time. And that's not necessarily a bad thing. If you go in with the expectation that you're going to do something really crazy and then get out, you can pace yourself for it and do something amazing in the time there. So once I had gotten Falcon 9 flying, once the team had gotten Falcon 9 flying, I felt like it was time for me to find another thing to do. And that was a bigger challenge. I didn't really want to leave SpaceX and go do something boring. And so I spent a long time, like, looking for the next thing. And during that time, I was working with one of my co-workers, Joel Brinton. And we had been working for a few years to develop this long coding recovery scheme, an algorithm by which you could recover extremely weak signals from below the noise floor in radio. And this has huge implications for things like GPS, cell phones, even submarine communications. And I myself have always been fascinated with submarines. And so this was an interesting area of research. And it was something that we were doing in our spare time, you know, weekends and nights. And it became very clear that we had a good idea and that if we really wanted to make this happen, we would have to quit our jobs and do it for real. That's when I had something that was worthy of my time at SpaceX for the next thing. And so I left and Ben Morse and I and Joel started Signal Laboratories to pursue this very fascinating algorithm that we had developed.

Jeff Ward

And how did that pan out?

 

Edwin Chiu

So the math went really well. The algorithm went really well. We developed hardware. We developed software. The business side of things was harder. And I left the company before we really hit a commercial market.

Jeff Ward

How was that startup experience compared to the SpaceX experience?

 

Edwin Chiu

Well, the thing about SpaceX is I started when there was about 450 people there. And when I started with Signal Laboratories, I was one of the three co-founders. That level of ownership is very different than what I had at SpaceX. You not only have to believe that you can find a solution, even if you don't know it yet, but you have to convince somebody else that it's a good idea. That was a new thing for me.

Jeff Ward   [11:24]

Edwin's social media sometimes features beautiful latte art. And I asked him if at one point he was a barista.

Edwin Chiu   [11:30]

I was a barista for a while, and it was everything I expected and everything I had wanted. It doesn't necessarily connect with the classic picture of building an electric bike or making a rocket. You know, this too is something that I got from SpaceX. I'm sure you remember carrying Higginbotham and the SpaceX Cafe. And, you know, the happiest part of my day was at the coffee bar getting my coffee in the morning. So I thought, like, wow, wouldn't it be great to be the source of that happiness for people, right? Because try as you might, do all the engineering you want. But it's very rare that a client will, you know, hold and cradle this electronics box that you've designed and hold it up to them and hug it and just be like, ah, this is exactly what I needed. You know, we did that for people at the cafe. We were right across the street from a school. So we got a lot of parents dropping their kids off in the morning and, like, getting their 20 minutes of peace with a latte. And it was a special part of their day, and I always treasure that.

Jeff Ward   [12:30]

When the pandemic arrived and put an end to Edwin's barista gig, he immediately started a non-profit.

Edwin Chiu   [12:35]

Yeah. So RespiraWorks is a non-profit that is focused on building a completely open source, free-to-license ventilator. There were a lot of such projects that popped up as a response to the pandemic. People saw how horrible the situation was with medical shortages, especially in developing countries. And we are one of those teams that got together to try and do something about it. And I think one of the things that was very obvious to us in the early days of the pandemic is that countries that have the ability to make their own medical devices, and specifically ventilators, would be obligated to supply their domestic needs first. And the horrible thing that we watched happening in front of us was that all of the countries that didn't have this capability were being left behind. And even if they had the money, even if they had the resources, they would not be able to help their own people because they relied on either materials from first world countries to make those ventilators, or the IP was locked up in manufacturing that could only be done in those countries. And so what we set out to do was to make a completely open source ventilator, which would allow those countries to use their local resources to make medical devices for their own people. So these are things that can be built with standard automotive and appliance manufacturing techniques, using, where possible, automotive sensors and things that don't rely on very specific medical components. And the big challenge with this tube is we're also trying to get it medically approved, at least in the United States, as a prototype, so that there is this kind of a baseline certification that shows this design can be certified for full ICU medical use, not emergency use, not temporary approval, but something that is a fully featured ICU ventilator that works exactly the way that doctors have been trained to use the vents that they would buy from first world countries. That would require very little additional training, if any at all.

Jeff Ward   [14:30]

How is the certification process going? How does it compare to, like, building a rocket that's going to be certified for having people put on top of it?

Edwin Chiu   [14:36]

Yeah, I think it's different, you know, when you're making a rocket, you're making a dozen or a few dozen of them per year, whereas, you know, with a ventilator, you want to make hundreds of thousands of them. And the liabilities work out very different. You're dealing with people, A, who are already sick, and B, like, way more of them. If you have a rocket, and you have a 0.1% fatality rate on a rocket, that's actually not too bad considering history, right? And people go into it accepting the risks of spaceflight. No one goes into a hospital expecting those kind of odds. It's very different, and we're still learning how to do it. And especially in an open source way where we're working with limited budget, we're working with people who are volunteers who don't necessarily have full-time hours to put into it. So it's a difficult thing, and we have a lot of contributors, and that's also difficult.

Jeff Ward   [15:24]

I asked Edwin if it was easy to get people to come and work on the project during the pandemic.

Edwin Chiu   [15:28]

The response was amazing. You know, we had three people, then five people, then 20 people. And within a few months, we had 200 people in 10 countries around the world helping us out. And is that your main activity right now? It is. However, I'm also contracting. Right now, I'm in the middle of a major contracting project working on instruments for creating genetic CRISPR-based treatments for people with cancer and genetic diseases.

Jeff Ward   [15:53]

I'm sure that SpaceX was about humanity for you, but it sounds like these other projects are a little bit more directly human.

Edwin Chiu   [16:00]

Yeah, I think it's part of a response to a lot of the human suffering that's been right in front of us. It's always been there, but I think that these last few years have made it really obvious. This past few years taught me that there was something I could do. That here in front of me was a contribution that I can make right now to help people out, whether it's, you know, making them a cup of coffee. And that's something I learned from the coffee bar at SpaceX. And I learned about this CRISPR work from my roommate who works at Synthego, who's doing CRISPR work and showing me that, like, there are things that we can do right now to help people. That's very inspiring to me. And also for me, too, it's personal. I, myself, am the proud possessor of a genetic disease. I have thalassemia, which is a single nucleotide swap. So just one letter in my genes and my hemoglobin is incorrectly folded and it reduces my blood's carrying capacity. And for people with severe cases of it, it can also stunt their growth, give them multiple organ failure and severely reduce their lifespan. And now with CRISPR and with Cas13, we can go in there, sniff that out, change it, fix it, repopulate the person's bone marrow with edited cells and have them permanently cured of thalassemia or sickle cell anemia or any of these kind of blood diseases. You know, those of us who grew up with these diseases, we grew up being told that there would never be a cure. There would be treatments that might make us feel better, that might extend our lives, but there would never be a cure for these things. And as of today, 20 real people have been permanently cured of sickle cell and thalassemia. So that's absolutely thrilling to me.

Jeff Ward   [17:37]

Sometimes we hear that CRISPR-based therapies are right here, and sometimes we hear that they're decades away. I wanted to know what Edwin's take was on the readiness of that technology.

Edwin Chiu   [17:46]

We are at the point where we are beginning to do human trials. There's still a lot to be done in terms of confirming safety. But for a lot of these people, the alternative is pretty harsh, too. But I think that the tools are there. We know that they work. And now it's a matter of making sure that they're safe and finding out how to develop them at scale. I know that a lot of the things that you do for pleasure and have done professionally are underwater. Did you get involved in submarines because you were a diver, or was it vice versa? You know, I had always been fascinated with the ocean. The space is wonderful and exploration is exciting, but the ocean, and specifically exploration in the ocean, has always been my first love. I grew up watching Discovery Channel, National Geographic, SeaQuest, just really being fascinated with the expanse and the infinite possibility of the ocean. In our lifetimes, you know, we will explore the heavens, and we will go to space, and we will explore the outer planets of our solar system. But it's very unlikely in our lifetimes that we will find life. And pretty much every time we take a deep sea dive, we find a new species. And the fastest way, I think, to meet aliens is really to go into the ocean. I've spent, I think, at least a week underwater in cumulative time over some 400 dives, some of which I did in a submarine that I designed and built.

Jeff Ward   [19:05]

What was the story of the submarine? Was that a commercial venture, another startup?

Edwin Chiu   [19:09]

Yeah, it was led by my colleague, Charles Chow. So he left SpaceX as well. And when I left the radio communications startup, it was around the same time Charles was posting on Facebook and saying, like, hey, I'm looking for someone to head up the electrical engineering for this submarine. We used to live together in LA, and we were building a submarine in our garage. And I basically called him up, said like, hey, you know, I think I might know someone who could do this job that you're posting about. And he basically said to me, like, look, you know that submarine we were building in our garage? Like, do you want to come do it for real? Over the next year, six engineers built a three-person manned submarine and deployed it in the Maldives and got it certified and operating.

 

Jeff Ward

Who wants a small submarine?

 

Edwin Chiu

Yeah. So this is the way I explain it. Trying to go to an aquarium and understand the ocean is like going to a park and trying to understand the forest. Right. There are so many things that are interconnected that you really can't see unless you go to the life that you're trying to see. And so instead of going to the aquarium where we've taken the fish and put them in a tank, why don't we put the humans in a tank and take them out to the ocean and really see what's going on out there? And what is going on out there? The situation is very dire. We have long viewed the ocean as this infinite resource. That is very much not the case. Our fisheries are suffering a lot. I'm working with another nonprofit called Mare Marine Applied Research and Exploration, and we do underwater surveys, and we often bring back bad news. Bleaching and coral disease are a huge problem. So there are some dive sites that I've returned to maybe only three years since my last visit, and they're completely dead. There's a site in the Mesoamerican reef called the Aquarium. And when I visited it, it was obvious. Beautiful, vibrant reef and fish everywhere, right? It was so dense. It was like being in an aquarium. I went back there maybe three or four years later, and it was completely bleached. The coral was dead and overgrown with algae and just destroyed. The Great Barrier Reef is more than 80% depleted. The Mesoamerican reef is healthier, but also feeling the effects. We need to understand a lot more about our oceans, and we need to do a lot more to save them.

Jeff Ward   [21:25]

It does seem like in Edwin's life, one interesting thing leads to another.

Edwin Chiu   [21:29]

An absolutely surprising turn of events. I was invited to be a contestant on a baking show, and I went on this show. Which show? It's called Baking Impossible, and it's streaming on Netflix. The premise of this show was that they would combine bakers and engineers to make these awesome engineered baked goods, like a cake that was also a bridge, or a cake that's also a robot, right? That can do something functional and mechanical or support structures and also teach engineering concepts. And we filmed the whole thing in Los Angeles at the height of the summer spike of COVID and managed to do the whole production without anyone testing positive or getting sick.

 

Jeff Ward

And do you have fun with that?

 

Edwin Chiu

Yeah, it was an absolute blast.

Jeff Ward   [22:09]

You should check out the show on Netflix, but Edwin's next comment gave away how he fared at bake-ineering.

Edwin Chiu   [22:15]

One of the unexpected joys of this show is that I've gotten to speak to kids about failure and how engineers deal with failure and how we use failure to learn, also personally, how we deal with failure. You know, we often think of success as this, like, unbroken chain of successes, right? When really what success is, is it's a lot of failures, the lessons from which pile up to a success. And often the moment before a success feels like abject failure. Like, I remember this one time where I made a pretty serious error in one of my designs and it caused a huge cost to the company. And I was so very sure that Bulent was going to fire me for this. And he said, no, I don't want to fire you. I want you to fix it. Like, I want you to come back here, get a plan together and tell me how we're going to fix this thing. I think that there really was this expectation that what we were doing was hard and that things were going to fail, things were going to break, schedules were going to get missed. And so it didn't feel like a judgment when those things went wrong.

Jeff Ward   [23:15]

Certainly at the top of the company, we did get judged for failure. But we also knew that failure was a critical component of our success at SpaceX. Fail fast and get the results correct the next time.

Edwin Chiu   [23:26]

I feel like I should get a t-shirt that just says in big letters, expect failure. I like that.

Jeff Ward   [23:31]

You seem like a happy and resilient guy. Did SpaceX have an emotional impact on you like they did on other people that I've spoken to?

Edwin Chiu   [23:38]

Yeah, it absolutely did. There are definitely ways in which it could have been better, but I don't think there are a lot of ways in which it could be better. The things that we were trying to do were just by nature hard. And sometimes a job is hard not because you're doing it wrong, but because the job is actually hard. Definitely. You know, I think it's good that I spent a limited amount of time there. I don't think I could have stayed there forever. One thing I said as I was leaving was that, you know, I would not go back to SpaceX now. But if I had the opportunity to do it again for the first time or to recommend to somebody to do it for the first time, I absolutely would. I like that distinction a lot.

Jeff Ward   [24:13]

I think that's a really articulate way of summing up something that I feel myself. The challenge of interviewing someone like Edwin is to ever bring the podcast to a close. And we had a lot of interesting discussions that ended up on the editing room floor. I realized that we'd run out of time, so I just asked one final question.

 

And what do you want to be when you grow up?

Edwin Chiu   [24:30]

I don't know if it's really a thing to be, but I want to explore. My goal in life is to see new things and to push forward my own understanding and the understanding of humanity. Whether that's in space or on Earth or in the oceans, is to go further and learn more.

Jeff Ward   [24:47]

Well, I'll be sure to check in with you in a little while and see what else you've decided to learn and do, because what you've done so far and described to us is certainly amazing. Thanks a lot for being on the podcast with me.

Edwin Chiu   [24:58]

Thanks, Jeff. This has been a lot of fun.

Jeff Ward   [25:00]

I'll put some links to Edwin's projects on my website. And I happen to know that as of November 2023, he's working on methane capture from abandoned coal mines. So there's always something interesting going on in Edwin's life, and maybe we'll check in with him again later. I hope you enjoyed the podcast. The reason that it's been so infrequent is that the interviews have been long and the editing process is very time consuming. I'm hoping in the future to focus on shorter interviews, make the editing easier, and make the sound quality better. If you have any feedback, positive or negative, please let me know. But for now, remember, most of the time, it's not rocket science.

It's Not Rocket Science · itsnotrocketsciencepodcast.com