Tung Nguyen, former VP Engineering at Bleacher Report

“When we go into the office, we’re given a problem that we’ve never seen before and we have to go solve it on the spot” – Tung Nguyen

About Bleacher Report

• 2nd biggest sports site in the world
• 80M unique visitors per month
• 1B page views per month
• 4.5 B push notifications per month
• 35 full time staffed engineering team

Welcome! This is the first podcast as part of the Devibe podcast series! I had the pleasure of interviewing Tung Nguyen a long time friend and colleague.

Welcome! This is the first podcast as part of the Devibe podcast series! I had the pleasure of interviewing Tung Nguyen a long time friend and colleague.

Tung Nguyen (Twitter @tongueroo | Github  tongueroo )  is an elite software engineer who as first technical hire at Bleacher Report, led and executed a technology roadmap supporting the company’s growth from an obscure sports blog to one of the world’s most popular sport news sites today. He has a tremendous amount of experience using cloud infrastructure like Amazon Web Services (AWS) and a wide skill set in using open source tools to quickly scale.

We covered everything from how “Lebron Decision” presented a major technical challenge, how bad 90s electronic music set a precedent for focusing in a noisy environment, how to get things done working from home with four daughters, how important mentors are in shaping your career, morning habits that power you to get things done, why Richard Branson wasted his time in a hot air balloon instead of witnessing the birth of his child, and so much more.

Enjoy the full podcast to immerse yourself into a well rounded discussion on how to architect your stack for massive scalability, home balance, nutrition, and the future of software engineering.

If you’re looking for a quick hit, skip to this highlights:

• How humble beginnings helped Tung to achieve so much more 13:00
• How a 9 month hard knock job got his foot into the big league 14:00
• First experience coding 16:00
• First day and first hire at Bleacher Report 20:00
• How to make the a career decision and transition 24:00
• Expanding B/R at scale 25:00
• Solving engineering challenges caused by the “Janet Jackson Halftime,” Super Bowl power out,” and “Lebron decision”  26:00
• Mitigating risks by benchmarking traffic 30:00
• Testing on production on a static S3 site vs staging 31:00
• Finding and addresses bottlenecks 35:00
• Architecting for 4 billion push notifications 36:00
• Tools and technologies including Redis, MemcacheD, Rub, Node.js, Amazon Web Services and much more 37:00
• On the challenges for building a cloud agnostic platform 38:00
• Must have services on Amazon Web Services 40:00
• Naming conventions and policies for building 45:00
• Handling failure: learning from success vs failing fast 47:00
• Handling the NFL crash with a slow query 48:00
• Brutal honesty and doing better 50:00
• Team building also involves after hours relationship building 51:00
• When engineers speaking to end users make sense 53:00
• Goal setting and payscale 55:00
• Lessons learned from terminating staff 56:00
• Will microservices solve all of your problems? 59:00
• Creating an environment to get stuff done 1:04
• The dark side of coding: dealing with constant change 1:08
• Impact of old age: how to keep up as you get older in your career 1:09
• Favorite books 1:11
• Eliminating distraction and reigning in the focus 1:13
• How a house of four daughters influences music choices 1:14
• Office vs home environment to get things done (home with 4 daughters!) 1:17
• Importance of mentors and the how to build a skill set with them 1:20
• Is waking up at 5:00 a game changer in getting things done? 1:23
• Balancing nutrition and sleep to perform better in software engineering 1:27
• Making time time for kids 1:34
• Effect of hacking elementary English by learning characters by site vs phonetics 1:35
• Challenge of prioritizing family, kids, and work 1:38
• Separating work from home vs. going all in 1:41
• What was Richard Branson was doing what while missing the birth of his child? 1:42
• Who wears the pants in a house where Tung is the only male? 1:45
• Severless architecture and containers 1:52
• Devops engineering success in the enterprise is a cultural or engineering challenge? 2:04
• Advice for beginners: you already know more than you think you know 2:06
• The benevolent dictator and the commitment for life 2:09

A books mentioned in the show

• Growth Mindset
• Never Eat Alone
• How to Win Friends and Influence People
• The 7 Habits of Highly Effective People
• Losing My Virginity: How I Survived, Had Fun, and Made a Fortune Doing Business My Way

 

—

Full Transcript 

Cameron Peron : Hello, and welcome to DEVIBE podcast. I am your host, Kevin Prone and here I decode engineering experts behind some of the most complex high performing and scalable stats to find out what analogies they love and use, what challenges they face and what the real life is like behind the curve. 

Today, on the Devibe podcast. 

Tung Nguyen: I tell my wife, ok what we do is we go and stop this and we are given a prop that we have never seen before, and then you just (inaudible) it. As we grew more mature what we have to do was we had to actually thick less risk. Faster for me was to rebuild parts of this than it is to kinda try to sit there and try to build a system that’s configurable to every single possibility. I wish there was a school or a book or a conference that you can go to that can teach you this but I am sorry this maybe this, this is tough love but you just have to live through the experience. 

Cameron Peron: We are here today, Bleacher Reports, the main office on market street San Francisco on a wonderful Friday afternoon. Bleacher Reports, Second biggest news sports site in the world with 800million unique visitors, 1billion page views. And we are here with Tanwin.

Tung Nguyen: So I have to go all the way back, I am gonna go to second grade. Umm second grade was in Stockton, California, and my parents had decided to move from kind of a more poor area to more rich area Lafayette, California. The difference in the education in the school system from Lafayette to Stockton is black and white. Stockton California’s education system is not the great. 

Cameron Peron: Lafayette is its the suburb of …

Tung Nguyen: Upper middle-class suburb..I think like 5% Asian. I am Asian and 1% black. It was extremely white suburbia, extremely rich, extremely well off. It was a brand new world for me, I walked in and I kinda remember second grade, I have some memories there of second grade. I don’t even know If I knew the ABC’s that well. I was just not that well off, not educated, I don’t think I had the vernacular skills, I dint had vocabulary to pretty much keep up with anybody else there and the reason being was because my parents or our first generation in immigration, first generation immigrants they came over here and then they just are gardeners, they just do what they can or make a living and they decided to move to Lafayette because thats where the business was. 

Cameron Peron: Okay..

Tung Nguyen: So,I was just kinda luck in there but I was kind of the poorest child in this very rich community. I remember this very distinctively because I kept on burning the midnight oil when I was young all the way to middle school like 7th 8th grade eventually before I got kinda good grades, It took me a really really long time to endure here. Even through my high school, my friends would come back from summer vacations and tell me ‘oh,I went to Europe’, oh I went to Disney land, oh I went to Hawaii, What do you do for summer? I mow your lawn, Thankyou very much. That’s what I did. okay. Thanks for telling me about all these great vacations because that’s what I did and that’s what I think cultivated kinda the tenacity that I always kinda had to driven in and work hard. My parents are Asian, so they tell me very stereotypical Asian advice, which is Son, work hard and you will be successful, Its really kind of typical what Asian parents tell you and I was having conversation with my buddy about a year ago and he was kinda bent into me and he was just saying like yeah my asian parents told me that too and I am like why did they tell you that because you know he was _____, I have been very successful in my career and we will talk about that later. Hes done more than 4 or 5 different startups most of them 2 or 3 years only old and they dint kind of pan out so hes kind of vent about this and for me as an Asian parent myself now, 3 daughters myself, I will tell my kids that ‘its not just work hard, and you will be successful, work hard so that you are ready, prepared so that when that door of opportunities open you walk right through it, thats what I will tell my kids.

Cameron Peron: Do you think that by being raised by immigrant parents that they kind of like pushed a maybe like a harder edge or harder standard for you to grow up to because they had invested so much into just getting here in the first place?

Tung Nguyen: I think definately, My parents made alot of sacrifices coming over here, they took a lot of risks. I dint feel all that risk. I think that was part of it. I think just being with the background and the environment that was around it and kinda seeing other people around me you know with the (inaudible), and me I will tell you a story. Every single year in elementary school for Thanksgiving we would go to school that week and we would be the winner of a free turkey, ‘winner’ every single time I say that. As a child oh mom, guess what we won a turkey. But my mom she knew in the back of her mind that we dint win a turkey, they gave it to us. So thats how they felt about us as a family in the community, thats how we kinda were, we were obviously one of the poorest families in the community. So, i think thats what kind of gave me the passion to drive, helped me define who I am and then eventually I went to high school, I got good grades and electrical engineering degree UC davis i went there and I got my education there. In school, i was a nerd all i did was kind of work. And in school, I was a streotypical engineer much different person than what I am today. My experience through kind of like my career my life has really changed who I am today. I can’t imagine what I was like then and what I am like now. I went through this ordeal of that I had to go and in college, I was young, naive and theres some benefit of being young and naive you just kinda go for it right and with without all the thinking and thats what I love and I am still like that to a certain extent but you also kind of as you are older you appreciate all the diversities that you have to go through in order to get to where you at. So, in kind of looking back at how I have gone, I went to electronic engineering school and than I work hard and get graduating, after I graduat I go to college and I got a job as an electrical engineer in Silicon valley. I did thats the dream

Cameron Peron: Does that mean, like just to understand it what that means, you were like fixing like electrical related issues like a building system.

Tung Nguyen: completely, hands on I am building heater circuits, I designed heater circuits for conductor machines. 

Cameron Peron: Great profession btw. Its a great skill to have.

Tung Nguyen: Its a great skill to have, I enjoy the labs you know like in electrical engineering college and all that. But the funny thing is and this is why when looking back I encourage people when I talk to people who are in college that go do the intern so you actually know what you are getting yourself into. 

Cameron Peron: So true, 

Tung Nguyen: And you dont know how important that it is until you go through it and you are like okay. Me, a week after 

Cameron Peron: Take the car for a ride, before you buy it.

Tung Nguyen: of course, and especially when you decide your professional career right. I became an electrical engineer and the first week of my job I remember I went to a to go sushi restaurant and it was not good sushi restaurant and how do i know that because I remember flies flying around and usually that’s not great indication of freshness in sushi. But just a to-go sushi place, I am just trying to get something to eat and then go back to work right. I kinda looked around as I was standing in line I saw other men in their 40s and 50s and some gray hair, glasses on, wearing long coloured sleeves button up shirts, khaki pants, penny loafers and I don’t think they were —- protectors but I imagined this in my head, and I saw a reflection, like a 

Cameron Peron: hold on, just for the context, what year was this?

Tung Nguyen: This was 15 years ago about so..

Speaker: 3 so 2000 2001 like crisis…

Tung Nguyen: year would be 2003, after college, ancient history, long time ago. It’s amazing how fast time flies especially when you are working and enjoying what you are doing. So I just kinda saw a reflection of my future self and I was like wait a minute what the hell. I have been working my ass off for all these years, what did i get myself into, this is maybe not what I want. Right? so guess what I did? I kind of worked hard again for two to two and a half years and I started saving. Coz i was like okay this not what I want maybe but I don’t hate what i am doing but I am gonna continue doing it because it’s not bad profession then I started going to Barnes n nobles for you know people today, what that is, it is a bookstore, there used to be these things called bookstores, yes they existed couple years ago where you go and actually read books. So I went there and read books about how to program websites. So I read alot of books then, i read it for a couple of months. So i was like okay, you know, I can do this, this is not really doesnt seem that difficult like I cant do it. So I saved enough at that point of time and I think it was like 2, 2 and a half years, Imma go for it, I am gonna go out there and start a website. My first website that I have built, started making money was a hot wheels website. I saw some others.

Cameron Peron: How fitting, I love that man

Tung Nguyen: Exactly, I love hot wheels, i can do it so i did it. Complete in over my head, complete you know out of my league enjoyed every singlee second of it. 

Cameron Peron: Now this was in 2003 2004, 05?

Tung Nguyen: 2003, 2004 yeah. So I did that and continued like 8 9 months almost a year.

Cameron Peron: Did you make money from it?

Tung Nguyen: I made revenue, I broke even

Cameron Peron: It was like referring…

Tung Nguyen: So you basically set up a bigger affiliate network, commission junction. I broke even

Cameron Peron: Sounds familiar

Tung Nguyen: When you are kind of going out like independently by yourself there, it is extremely difficult when you dont have money coming in right? So i ran out my savings. What am i gonna do now with my life? Should i go back to electrical engineering, i have couple of years of experience doing that now so I would easily get a job there and I have contacts in the industry and that’s the safer route, common sense route. Or should I do software engineering, I kinda like that too? I don’t know what I wanna do, maybe little riskier. But let’s see what shall I do. So what I actually did was I went out there and I met a recruiter, at this point of time I had also picked up ruby on rails, it is 0.2 days what happened was I told you I read a bunch of books on PHP, I did not like it, so I said I am not gonna do my website in there I am gonna build a pearl and did something that’s — that’s gonna be oxymoron, object-oriented Perl, this makes no sense, so basically you have to jump through hoops in order for this language to behave in object oriented way but I actually did it. I learned it, I went and did all that. And then what happened was I ran to DHHS 50min blog, infamous still on the internet today and he just showed how he can use this framework in ruby language to build a blog website so quickly and under 50 minutes. I sat there and I thought to myself for 30mins like I am not gonna really do this, I just spent months of efforts learning this thing and I am gonna rebuild this right now. 5days later done. So, that’s when I was like wow, I think I have really found a programming which I love you know and something I like to do and that’s actually extremely productive right? At that point of time, that’s kind of near me having money so I was like okay let’s find a job. Then I talked to a recruiter, she lined up an interview for me that afternoon right after lunch, I got a job offer in hand in the email that night and I started 2 days later. 

Cameron Peron: Which company was that?

Tung Nguyen: This is (inaudible labs). This was 10 years ago..So you know and kinda going forward to living it here because I am a bleacher so that’s my first start up. I was the 3rd engineer at that company, I helped built that software platform. They eventually sold successfully.

Cameron Peron: So hold on like, in order to get to (.. labs), did you look at any other companies or was it kind of like you call the recruiter and she found you a job, you got the interview, they said yes and you just went for it?

Tung Nguyen: Well, yeah I kinda just went for it, but week prior to that what I was doing was I was sending out my resume to companies and jobs I was looking for, I was looking for job descriptions and kinda tailor my resumes specific to the job description because eh this is a fit and basically I was looking for companies that I wanted to work with. Basically looking for Ruby and rose engineers. There weren’t that many, but it’s really interesting. Engineer managers this seems like now I see it on the other side of the fence because I have been through the engineer management phase myself now. They don’t have a lot of time to go out there and like scan through all the resumes and craigslist and all that so you really kind of outreach to them and then you will get response and I got some responses but alot of them were like okaay..you know not enough experience.

Cameron Peron: You were reaching out to the HR people in those companies? Or you reached out to like the engineers?

Tung Nguyen: I reached out to whatever was on the job description so most of the time was HR. So i was not getting through that filter.

Cameron Peron: This is back in the days, before LinkedIn?

Tung Nguyen: Way before LinkedIn, And you know I had a good tactical strategy trying to get that job. Right? So that dint work but once I met somebody you know who was an engineer I got in right away. Why? because of that 9month of experience that 9 months of chance that I took was an invaluable experience. Like there is nothing you can gain.  I wish there was a school or a book or a conference that you can go to that can teach you this but I am sorry this maybe this, this is tough love but you just have to live through the experience. 

Cameron Peron: So what effectively happened here is, Is you went to school as an electrical engineer, you learnt how to learn, so it sounds like you got there from your parents you got this from school then you went to Silicon Valley to do something completely unconnected to tech with all due respect. And you said alright I am gonna figure out how to make it here so you built your own website which could have been like a business. You kind of like funded your own education in some way, you learned the ropes and you kinda created an internship or kinda of a job for yourself that brought you some kind of credibility and you got a job at infinity labs. Right?

Tung Nguyen: yeah

Cameron Peron: Amazing, As amazing it sounds very practical way to do it right? Just a second, We talked about umm we talked about you building this site, was that your first experience coding? 

Tung Nguyen: That wasnt my first experience coding. My first experience coding was actually in college, where I did some C programming, I did some assembly program and the issue of programming in college is they are usually a couple of years behind especially back then before they invent the internet and all that. They  were many years behind, I think they were teaching seeing all the companies were using java. Nowadays, I see universities teaching reals or node kind of a newer training technologies here so they are keeping up workers to be enforced to because the internet is moving so much faster so thats great but they are always gonna be a little bit behind. The exercises they give you are very academically based, so they will give you a problem to solve that honestly doesnt really bring a real vibe. So this is for me a, I like to be productive. I will tell you a funny story, A couple of years ago this was me at that point in the career at bleachers I was doing more management stuff. At that point of time I was kinda busy all day, And this one day I was where I was hit left and right with distractions, It was just one of those days where it was unbearable, I got nothing done, at climax at 4pm where one of the exercutive assistants called me over, she was the executive assistant the CEO at that time, I love her to death so right when she emailed me, text me and called me all at the same time, is emergency come over and I said ofcourse I rushed right over and I was like whats going on, tell me whats going on. Okay..so this got tossed on my lap and I have to deal with it. We are upgraing the internet service at the company right now today in the building over this weekend is it going to crash the site? I am just like

Cameron Peron: Okay. Just for the context. What year was this?

Tung Nguyen: This was about 4 years ago. So I look down on the ground

Cameron Peron: Small administrative decision..just change the internet service..what could possibly go wrong?

Tung Nguyen: But for me thing is..I was like I kinda look down is let me explain this to you..This building everything in it can burn down to the ground ashes and all. And the site would still be fine because our servers are not located, no servers are located in here. All located in data centre in west virginia. Now its not gonna effect it what so ever. So i went back to my desk and before going back to my desk I may have stopped at our snack area and I grabbed a jumbo size snickers bar, I grabbed two jumbo slimjims, I think I grab a bag of chips and I kinda sit down 

Cameron Peron: Gret nutrition man

Tung Nguyen: And i think there were couple people around me still and designer nearby she just like look disgusted at her face, this other designer just bust out laughing, this other kind of engineer nearby he just like a pole shaking his head. And i was like this is what happens when I am not feeling productive. So, I asked my friends to do this favour like dont let me die and pull my own donuts right? I like this feeling of productiveness.

Cameron Peron: So you were infinty labs? What was the course between infinity labs and the bleacher report?

Tung Nguyen: Sure, So from infinity labs to bleacher. I have been working for infinity labs for a while and the founders actually approached me. I met the founders through one of my buddies, he was at a coffee shop. Streotypical story, coffee shop, eating a burrito with a ruby book on his desk and one of the founders walks by eh you know ruby and a guy who knows ruby. You wanna work for me? Nah i got my job back and the introduction was made and then I had lunch with the two foudners and the conversation went pretty simple.  I asked them do you guys have any money? And they said no we do not. I was like thankyou so much for your time. Thankyou for lunch, have a great day, goodbye. That was end of the story. And then what happened was I went back and continued working. 2 or 3 months later, my boss at the time got some funding VC to go start his own company, he flew back to canada. Basically he told me goodbye, bastard, just kidding, I love him, good guy. He left and I was a little bit bumped so I went online and was like okay bleacher lemme send a registry for account and within 30mins I get an email from one of the founders I had lunch with and the email was like Hey tun, he dint say this but basically I have nothing better to do at my time other than look at every single user registration that close to the site I notice you register, you have change of heart and lunch again? I was like are you kidding me? Thats like little bit creepy. But at the same time I was like that kinda cool.

Cameron Peron: Kinda dedicated 

Tung Nguyen: yes exactaly, so i was like yea sure why not. So i went ahead and had lunch with him and you know they had some kinda of cold war to see whether or not they like me, funny story they told me later. But any ways, conversation wewnt like this. I asked them do you guys have any money, and they said yes we do have money and I am like why not? Lets just go do this, these guys seem smart, these guys very story coincidences here. Then i kinda started working for them and the first day at the job empty desk and nothing there. I kinda went in brought my own laptop and they were like heres your empty desk and goodluck, we have to go now. I was like where do you guys have to go to? We have to go get money, I was like thankyou very much guys.

Cameron Peron: So hold on, you came to bleacher report, how many poeple were in the company when you joined?

Tung Nguyen: It was just the fore founders and me at the time.

Cameron Peron: wow, so you are the first technical hire.

Tung Nguyen: I was the first hire period, so when I started it was me and the fore founders. They had two contrators already and the were building the site using ruby on rails so I inherited this kind of rails cobase and I started modifying, started goingn with it. I told my wife at that time like this is gonna last like 3months but I am gonna learn something, and this might be fun but you know it just kept growing over the years like I started of this lead developer then it was back engineer manager then it was director of engineer and then it was senior director of engineer and now it is VP of engineer at bleacher report. And I have seen this whole transition from what angel series to series A to B to series D to E, Series C, Series D acquisition, post acquisition. I have seen the whole life of life story of company’s history. I have lived through then. And then again you cannot learn that out of a book I dont think, theres alot of experience along the way that like I dont see any other way how could you learn that unless you kinda lived through it and I know this changing me as growing me in many ways that I never thought would be possible

Cameron Peron: Yeah, The practical knowledge like trumps any kind of academic knowledge 

Tung Nguyen: Completely

Cameron Peron: In infinity labs, your boss like left so you werent comfortable working there any longer, you were thinking about a new gig. Was it something specific about bleacher report that you’re tuned into? Its a sports website you know, you know sports?

Tung Nguyen: I watch a couple of sport, I watch football, I watch basketaball. Its not like I was that.. I did say I kinda like sports and more importantly I like these guys..you know I thought these guys were smart and I thought these guy kinda grew up in a good area, Paul auto and I was like with their background this better odds so I kinda go with them so i just went for it. I will like say it like oh no, I like evaluate them i looked at their sries no no..now i know them more, now i know them better, much coincidences right? Thats what it was for me.

Cameron Peron: What year was that you took the job?

Tung Nguyen: Its 2016, so 9 years now so 2007 2008 so it was 2008.

Cameron Peron: Yea i remember 2010 was when bleacher report got kind of got around yeah,

Tung Nguyen: Yea we have grown every single year so we have been very very lucky in that sense. 

Cameron Peron: So how did you take that? Being some sports blog to being the second biggest sports site in the world?

Tung Nguyen: Take that? This is great! I will take it. 

Cameron Peron: No I mean how did you do that? Its not like a small trivial thing to be managing behind the scenes. Right?

Tung Nguyen: ofcourse, of course not. I mean you expand as you need to, you keep on growing along the way. I dont think there is a special magic pill here right? There is no secret answer. It all depends, for us our traffic our patterns are statistical, basically, there are peaks for football season, peak basketball season but there is a sports calendar there. So what you do is you these give us, gave me and our team learning opportunities along the way the whole way. Every single time we had peak period things would break things would break and we would learn from it and fix it and then kinda we move on. We kinda grew, we had peaks we had events to spice, but we just basically kinda kept up along the way, its hard to explain,, there is no one way answer to kinda that, looking back now, i know that wow we should have done this and that but along the way we are just like okay what is the pain point and lets just solve that one or two specific pain points and then lets move on to the next challenge.

Cameron Peron: What was the some of the craziest back in challenges that you had to face?

Tung Nguyen: We have had all sorts of crazy sports events from Janet Jackson exposed her nipple right? So this was ridiculous, so once that happened I guess everybody jumps online and looks for sports content about that and we have couple of articles about that so that blew up our servers. So we had to write additional cashing right there on the fly, on the specific article, coz thats all you really need to do. Theres been other events, the following year the superbowl half time there was a blackout, so that drve traffic again, there is something else this year, our biggest day traffic is still by far was Lebron decision, I dont know if you guys remember the Lebron deccision. For me, I am a sports fan but like these things were really big, major events in our engineering life because of the traffic it drove. Lebron’s decision, He said the whole time ‘not a big deal’ but ofcourse they had the whole hour special on tv about him. Right?

Cameron Peron: I wasnt living in the US, so what was going on exactly?

Tung Nguyen: Sure, his decision was like he was playing Cleveland cavaliers, he was deciding whether or not to leave the cleveland cavaliers coz his contract was over and to move down to miami to try to win some championships, which he did, they had a big special night and when that came out, he announced it. We prepared up by basically looking at very basic things, people think like alot engineers kind of think scaling and making sure your site cant handle the traffic is some kind of a magical thing thats very difficult to do when it actually comes at very basics. Right? If you have catching, you are pretty much covered really and then you have to go back to basic. Slow queries log these are things that cant be learned that usually people unfortunately learn through pain but we learned it through alots of pain. But once you kinda learn it you just make sure not to do it again. There is no other way to put it. Or you learn how to degrade your site, it kinda flip out portions that are causing issues immediately. So we went through this rapid transition where we had to do was okay there was a prom, lets basically disable that part, and then that will buy us time and we will go back and fix it. It is very hard, when you are a startup, this is the life you live. Right? As we grew more mature what we had to do was we had to take less risks. Facebook, I think says this that move fast and brake things. They dont really mean that, its a catchy slogan. But what they really mean is move fast with mitigated risks. and as you grow bigger what you have to do is you have to mitigate those risks. So as we got bigger what we started doing was we started benchmarking. So you throw a chat(inaudible) that thing and you know what the pain points are and you are still not gonna catch everything thats why you need to know enough about the system and you need to be able to move and deploy fast enough to fix things when emergencies do come up. You are kinda have that reassurance if you have benchmark kind of earlier on.

Cameron Peron: What kinda benchmarks have been important to you, if you could kinda frame them over like the nine year peiod that you are describing like I am sure they changed but how they looked like?

Tung Nguyen: We have used every single benchmark until I can remmeber, like AB I have experimented with these are very specific tools right? you have tools that are specifically kinda test one server and test whether or not that server is gonna overlaoad and how many requests per second, then you kinda know what your application can handle. Right? Thats the kinda baseline that you kinda go from there then we kinda test what kinda cloud based tools like blitz where you kinda throw massive amounts of traffic and its very important use somthing thats more cloud base where you can generate traffic and test something and you actually do it on the production site. We do it at quiet hours at night so then we know where our stress points are, we have done this with stagions too but its not the same. Stagions can give you a relative `measurement but its not going to give you the absolute measurement to know when you are going to die. So what you do is you test it on the live site when you can and then then you know where bottlenecks are and then you go back and test them again. I think alot of people when they hear oh this crazy who tests their production site and all that. Well guess what we have safeguards already, if our site crashes over we fail over to statistic s3 site. Right?

Cameron Peron: Wow..statistic s3 site.

Tung Nguyen: yeah. We degrade and this happens automatic fashion. I remmeber when we first implmented this there were questions about if we fail should we automatically fail backward or not like no you always suppose to have somebody press the button. No you are not gonna be a machine. A machine could be automatically faster then a person so we automatically fail through route 53 health check and then we automatically fell back when we are healthy again. Why? coz by the time somebody answers their page system has already healed us off. And then its outer scaled up so its not you are gonna beat the machine. So that whats we have done we have put in these safeguards systems so then we can do these things peoople consider extereme, but i dont consider extreme at all, I consider this is how you actually know and thats just how i have always operated. 

Cameron Peron: With the janet jackson thing that happened can you describe like what exactly happened and how you solved it?

Tung Nguyen: I dont remember the exact details i just remember that there was a ton of traffic then we had to catch that specific article that was being hit all the time. So thats what we basically did there to kind of solve that. All that was really needed to do. Its i think alot of people thing to make everything fast right? you are not gonna have time to make everything fast.. Focus on what you actually need to make it fast. Kepp it very simple. We, I had alot of process that i had to precatch like popular pages all the time and that would take care of most of the cases.

Cameron Peron: How did you architect that?

Tung Nguyen: Well the architect behind that its not complicated architect, all you have

Cameron Peron: It isnt complicated architect but when you are getting that level of traffic it is a bit complicated right?

Tung Nguyen: yes ofcourse there is alot of moving  pieces but you only focus on the highest throughput. In order to focus on the highest throughput you that kinds of finds out the most popular pages and make sure those never hit the sale catch..never ever. Infact, one kind of tricks that we have done is we kind of pre-brick lot of our pages and with very very high ttls. That means that they never get naturally generated when they get naturally generated. Nowadays we have more safeguards, for a while they were naturally generated it would hit to database directly, you would have all the requesting the same page at the same time and regenerating everything at the same time and it will overload the system. And you can kinda scale out and we have done that we have done that kinda additional slave servers. Its not as complicated as people make it out to magically be,  you just have to find your bottlenecks and address those specific bottlenecks and once the database is ovferload we fix the sloq queers or we add slave tears and then you have to make it highly available low balance replicas. If any replica goes out of commission it still serving traffic, the whole proxy from those low bounceres is great but i dont know i just thought alot of  people think that you kinda have to use these newer technologies all these new things,  I have toyed around with lixer with go with many different languages and many different firms we tend to enjoy alot of them but i think with most cases even as site as large as bleacher you know you can get away with getting any application that is 200 milliseconds response time and you know thats fine and you can do that with ruby most of the time, you cant always do that. One of our  applications where we kinda had to revoke that was the push notifications aapplication what you mentioned earlier. 4.5 billion push notification, yeah thats alot, that one we had to re architect coz that one required high level of paralelism it was different type of processing, it wasnt like a request that you could not catch, it was a dfferent type of scenario there. Its basically pushing out bunch of alerts to everybody as fast as possibl, In that scenario what you wanna do is you wanna start pushing those alerts right donw and paralyze everything. I dint rebuilt this, alot of engineers rebuilt it, Eddie, Victor and bunch of guys rebuilt this and when they rebuilt it they use a different architecture that made more sense for that exact scope of the prob. But they dint necessarily went back and changed everything, they solved one specific problem with that technology that they needed and that why engineering its not really there is not a cure for all this you gotta use the right tool for the job basically.

Cameron Peron: CAn you describe some of the tools and technologies that you are using in your stack?

Tung Nguyen: WE use kinda everything, we use ruby and rails, steelonet, elixir, finix, trending technology right now, we use nojs for some of our components right now too. So we kind of use those three very trendy popular frameworks. Thats the language of the frameworks we use. Terms of catching, mencatching one of the most stable things we have ever used, and then on a big day on the analytical side we use redshift and things that make sense on big data. Again you wanna use the right tool for the right job here, but we use a alot of every other thing just depends what we need.

Cameron Peron: Is that all on top of 8ws? 

Tung Nguyen: This is where i got lucked out, got fortunate maybe coz i was around for so long, so we used 8 of us for a long time now. I have been on on it for 7 71/2 years ago.

Cameron Peron: They started in 2006? Earlyy adapter

Tung Nguyen: before they had alot of tools than they have now

Cameron Peron: Was it still s3?

Tung Nguyen: s3 easy 2, they dint really anything else, and when i started using it, i dint use it by myself i had another company helping us at that point, NGR, and at that point helping me learn about that. ALot of people would say that big companies say this all the time, I wanna build a platform, a pass and i want to be called agnostic, i want to work in google club, ibm rock space, i think this approach, in my opinion, is very difficult to already build oppose and whats gonna eventually happen is you are gonna build a kind of platform but that platform needs to work for all of these vendors and what results is non mary platform. you basically have to use whats available for everything there and if its not available then you build it. so what works is you build something thats not as great as something that you focus on one good vendor.

Cameron Peron: yeah thats a big compromise on what you can do and how much control you get and access if you use like a generic platform llike that and you could do on your own

Tung Nguyen: ofcourse, so this is the problem that i had to solve i know if vendors out there they need to go and build things for customers and sell it, but me i dint had to solve this problem. so for me my problems were much easier i just went to autobass, and all the clouds were catching up and they will catch up and it made sense, what i ran into was they had all the tool ready all you had to do was write bunch of tooling on top of what they had and you wrap their tooling and you basically have what they have. And you just have to build alot of tooling between in there. That approach worked really well for us and are off teams need to grow now but it was really small for a long time, Me i was only abscound on the team,  in the recent times i started doing ops. In recent years i did alot of of ops and recent 4 years i did alot of ops works, and i was pretty much only ops guy running pretty large flee of very highly available 

and i was pretty much only ops guy running pretty large flee of very highly available servers, 

Cameron Peron: What kind of aws services are you using?

Tung Nguyen: We use all of them i mean, we use elb, ii think very important service that we use the cloud formation, this is probably not very new for alot of guys, but for me when i was building um what we call bear cloud or our version of tooling on top of 8ws cloud, when i started off doing that I was basically rebuilding our vendors solutions, i built it gooey, provision service, i went pretty far but it took me 3months or so. and i was getting or moving as fast as i like and I even pulled  a developer to help me in this and he helped me in this then we got it deploying code we got it provision servers so we got it running shelf what we happen to use there we got a low balance rate that time working but it took 3months and then i had a drink with a buddy of mine whose a ops guy and i kinda complained to him like okay i wish i could build this faster i have so much to do and he brought up to me at the time cloud formation like 3 4 years ago and i was like yeah i gotta finish this thing i am too busy i dont have time for this.

Cameron Peron: I dont have time for somebody to help me do things better, i am too busy chasing my own tail

Tung Nguyen: He kind of smirked, oh great, this guy is a smart guy, and he knows what he is talking about and I was like oh great i missed something obviously so i spent that weekend i read through all the documentation and then i came back and i emailed him like eh come work for us, come contract with me, he came within 2 weeks time. He had done what i did 3months, i basically got one application kind of up, he got 5 applications up in 2 weeks. At that point i was like okay i am gonna really do this, i am throw away 3months of work, i was like yup done. Completely you have to do this. This is not the first time i have done this. Our codebase application, we built it many many time over. This is just the fact of life, if you dont rebuilt your own codes somebody else will do it for you. Infrastructure codes have been the same way, our infrastructure code, this is like the 6th version of it. I rebuilt i think 5 times.

Cameron Peron: What were the reason why you just not thought about it but did it?

Tung Nguyen: Did it like as in rebuilding it?

Cameron Peron: yeah

Tung Nguyen: yeah coz every single time you are building something you have an idea what you want, but once you kind of run through issues thaat changes it. Theres some people smart enough out there probably to have a vision of exact way they want like you know 2 or 3 year out and kinda know, no thats not how it works in real life. In real life you learn on the way, i am not saying just to be clear that you have no direction at all you have to have an idea of what you need right? But you want to bring that in from experience, from what you have built before and you want to apply those lessons very rapidly. Then you want to have an environment where you could rebuild it anytime you want and thats what i did. Then i started off with very thin sets of conventions, and those conventions actually exist today.

Cameron Peron: are those conventions built on a certain amount of time that you wanna learn in and redeploy or is it based on some kind of events or some kind of challenge?

Tung Nguyen: No those conventions are, well there are some ideas of what you want out of the infrastructure code or what you want the application code or what you …, what you wanna have is you wanna deploy quickly you wanna move quickly and all that. Everybody wants that. but the conventions are really kinda like, here are some naming conventions, here are some policies that follow in order to kind of build this, and these naming conventions kinda bleed through the entire system. So the naming conventions are (inaudible) certain patterns. You gotta have some kind of consistency here, this is not a new idea, convention configuration. What i found is alot of companies want to build systems that kind of cater to more possibilities, more configurations right? but what ends up happening is, a software that tries to caters everybody, no you gotta make a damn decision. You just have to go no this is the problem i am gonna solve and this is it. If i need to solve beyond that. Its faster for me to built parts of it then it is to kinda sit there and built a system configurable to all the single possibilities. thats how i approached it and then you learn along the way and then you kinda rebuild it along the way so this is, there is no other way to describe it, thats exactly how we built it. Then what you do is you see those conventions stay with you forever, its quite amazing. When other people join in they can see, once they learn this pattern exists here they see that the pattern exist in the entire system that bleeds through everywhere. 

Cameron Peron: nice..okay. Lets talk about failure. You have been in the picture here as a an engineer and a leader. Explain like issues you had or you failed and what you learnt from them?

Tung Nguyen: yeah. Failure..i feel, i have failed alot right? You just gotta bounce back after that. And its fine to fail as long aas you fail fast and kind of learn from it and i think thats a very common expression. Fail fast and learn right? I think a less common expression is in why dont you learn from your success? So you kinda have to do both. Learn from it but you have to be really brutally honest when you fail right? this is the thing that kept me going, When i make a mistake, I am actually pretty brutally honest about it. I am just like this is where we screwed up, I am not making excuses, I am not sugar coating it, we failed, we screwed up, I want us to do better. One of our biggest failings was actually NFL drafts, another big event. We failed terribly.

Cameron Peron: Which year is this?

Tung Nguyen: I think 3 years ago, we just basically crashed, we couldn’t handle it. because at this point we dint had benchmarking yet so what happened was massive amounts of traffic as we suspected but a slow queer struck in into one of our popular pages so then we had to go address that. What we did was I gathered guys in the room, I said last night was terrible, last night did not go went well, its okay, what i want us to do is treat every single hour from this as our site is kind of failing right now and you guys know the code very well. Find areas that you all know that could be problematic and fail switches in there, add basically feature flags, do whatever it takes. The guys gathered around and what they did was they did exactly that. They added the feature switches and found the slow queer hour before it happened, we fixed it and we had a day 2 day 3 black and white. Completely black and white. Since that day on we have always kinda had benchmarking before the big events. 

Cameron Peron: so the NFL draft event it crashed, you pulled the team in the room and had discussion. What were some kind of points that you helped to bring in the picture to motivate them? was it kinda like we failed lets implement some kinda strategy and paused everything or was it one kinda motivate the team and there was a sense of true failure in the room?

Tung Nguyen: yea ofcourse there was failure in the room, I am brutally honest with them so I just told them, I am not happy and i know you guys are not happy either but guess what we just gonna have to go and do it adn it wasnt, I dint had to go in there and tell them ohh some brave heart motivational speech, I was just honest. I think people dont appreciate that, you know people sugar coat it. People care about the perception of things then the reality of things.

Cameron Peron: Whats the age range in your team?

Tung Nguyen: Straight out of college, early 20s to 35.

Cameron Peron: And the brutally honest approach is a good one, getting to the point and getting things done?

Tung Nguyen: I think the brutally honest approach is good one for me because i have built up a repo with guys. I mean if you havent built up that kinda relationship with all your team members and you just go in there and you are like brutally honest and you can come outside looking like a jerk.

Cameron Peron: What does it mean like you guys go for drinks? lunch? friendly proffessional repoire? 

Tung Nguyen: Both. It depends on, some people wont go out for drinks, some people just want to have a conversation in the office. Some people want me to help them solve their problems. You put the time and commitment here. There is no easy way here. Engineering is a hard thing to do, I am an engineer myself I tell my wife waht we do is we go into the office and then we are given a prom you have never seen before, and then you just have to go solve it. Its really rewarding. I love this. This is like, i am not tryna say like discourage anybody because engineering is a really rewarding career. Its just that you built something and its your baby and you are proud of it and this is why engineers continue doing what they do, they like talking about the littlest things. So you just continue talking about them. 

Cameron Peron: So how do you empower technical decision making in getting them to execute without you getting in the way?

Tung Nguyen: So yeah, first of all technical decision making, i think you empower them, a new person starting a company, a new person straight outta college, they dont necessarily want to completely empower all decisions, they want help. So it really depends one power. more senior guys more experienced guys. Guy who wants more responsibility, you empower them right? you just get out the way. You just line them who ever they need to talk to basically the customer, the person who is using the tool. Lot of people they built tools for internal tools, you get them talking directly, alot of people built tool here for consumer, people who use the site, get them talking directly,

Cameron Peron: you have some of the engineering team talking to end users of bleacher reports?

Tung Nguyen: I try to. as much as possible. Not always. 

Cameron Peron: Is it working? or if it doesnt work whats preventing that from happening?

Tung Nguyen: theres always can be more as you go bigger you wiill always have a hierarchy. over the course of many years here we had a very small team at the beginning, now we have 35 engineers or so. And you have to have some hierarchy you cant just have flatness. I dont think you want completely flatness either. The person at the top is supporting everyone at the hierarchy thats how i see the team should be structured. What i try to do is you get a couple of players. 

Cameron Peron: I assumed that you have terminated some people at your team? Tell me what is that like? and what is like the lessons you have learned in how to do it respectful way?

Tung Nguyen: Terminating someone is never a fun task. When you have to go to aa length to fire somebody, I see it as it is a sign that i did the bad hiring. I take that on myself, i could have made the situation better. I dont completely blame myself because the process of interview 2 3 hours with somebody and you hire them to work with them forever its ridiculous. What you do is you try to grow that individuals. When it gets to a point when you have to fire them then you work on Cushing them out, you kinda sit there and go like maybe this is not the right situation for you, but how can i help with your next engagement? and thats how i have made it the mmost easiest but not all my cases have been like that. 

Cameron Peron: you coach them out the company or you had like direct discussions like sorry thats not working?

Tung Nguyen: yes i have that too

Cameron Peron: but i guess for the ones that are totally bad but just arent performing very well 

Tung Nguyen: those are extremely difficult to do, i engage as much as possible, i rather for me consistency is the measure, it is so important. you just have to be honest and do it in a very respectful way. In this market, engineers are extremely sought after, if you are not a good fit here doesnt mean you are not a good engineer. we look for a specific kind of an engineer.

Cameron Peron: What is impact of micro services in your work in your team?

Tung Nguyen: so buzzword..microservices. Everybody thinks there are elixir bolt solve all our problems. It comes with a bunch of complications. honestly when a request comes in there is a response, so what you do is avoid everything thats supposed to be a microservice artitechture. we try to avoid slow call at third party vendors instead we throw them on jobs. In microservices what you are doing is that you are doing nothing but having network calls everywhere.  

Tung Nguyen: Um if you keep the prom  kinda of independent  and it’s a separate kind of problem  with that the meritisal service system then that is when you kinda wanna break out um we’ve gone kinda of both ways we’ve gone slowly in microservices where you kinda monolith and then we should broken my service earlier and we’ve gone the other way where we’re like we broke it too much and i think we should start rolling back up right? so i hope to see uh bleacher and the team roll things back up a little bit more here now because you have just too many network dependencies and isn’t of having one big monolith now you’ve uh i think what Netflix refers to as a distribute monolith  or your complications are just distribute across the network 

speaker 3: yeah 

speaker 4: there is slowness everywhere so um its not like I’m not fan of micro service i like micro service i like the concept i like (inaudible) micro service architectures are built i think for larger companies a larger company lets say i don’t know thousand two thousand infura google LinkedIn whatever those companies they have lots of more resources you know right smaller companies have reality right uh with thirty-five endures we’re not small either right but you don’t wanna have like a hundred micro services cause when you do that again good luck keeping track of all the dependencies you kinda need you can do that i think its possible but you have to force pretty strict standards and pretty good um uh plot from for in terms of infrastructure you have to allow the developers to kinda do everything they would need to do without kinda any intervention from any other team if they can’t do that then its gonna be bad news

speaker 3: got it if you use anything in place of those um micro service specific (inaudible)

Tung Nguyen: so i just talked a lot of bad stuff about micro service but for whats worth we do have micro service architectures right now we went from one application to five um five years ago to we have about twenty different applications right well I’m saying is uh that doesn’t come for free thats all I’m saying so we still do micro services but instead of just saying oh we have problem oh we will just go service for that oh uh you know you have to go kinda actually make more sense to actually builder in the same application where our app or and then you the idea theres one and that extreme micro service is where everything is independent everything doesn’t talk to each others data base right and the other thing (inaudible) where theres is one monolith and you just keep adding  pervious monoliths right its you don’t have to go one end of the stream to the other end of the stream there is something in between there right so its okay for some services to talk to the database like and this is gonna be a pompler specially with all the books out there talking about oh these micro series should completely be independent I’m saying that makes a lot more sense uh a larger organization with thousands of employees right from the small organization with uh you’re expecting the engineer to know more about the system be able to kinda take his coat all the way out to production that makes thats a tough ask

speaker 3: yeah yup yup true ok how do you make a environment for yourself like to make things happen around you

speaker 4: for me uh my schedule is a little bit different but what i kind learn over the years is uh i try to kinda of raise my time so that all the things that require multitasking all the things that are distracting activities i try to do that in the beginning of my day and…

speaker 3: beginning or end?

speaker 4: beginning it doesn’t matter just group them all together you could do it beginning or end doesn’t matter it depends everybody is different 

speaker 3: what like email or?

speaker 4: email checking status updates all that meetings right all together and then the second half of the day thats the one thats un interrupting  

speaker 3: do you do it like like you do that in the beginning of the day and then like the second part of the day is like you’re focused you can’t be bothered you could be 

speaker 4: thats thats how i try to do it yeah i mean not always successfully to be honest but thats the idea approach and then you know uh usually once a week we have a day when there is no meetings at all 

speaker 3: yeah 

speaker 4: so i mean a lot of the companies practice this and that works because for coding and engineering tasks you require un inter of the blocks of time 

speaker 3: yeah definitely 

speaker 4: and if you don’t get that quietness you don’t get that focused you’re not gonna get anything done yeah and but you do need and assistance grows more conflict uh complicated you do need time where you communicate to uh your colleges and kinda figure out what the blockers are and kinda figure out what i need form this other era or this other discipline that i don’t understand and that when you uh thats the perfect time for the status mornings kinda the meetings and all that 

speaker 3: okay umm what are the hours of the day for you that are most effective between getting things done and like creating or synthesizing?

speaker 4: so because I’ve kinda of spilt out my uh schedule that way the synthesizing part has to happen second half of the day thats the quite period you need quietness in order to do that i see no other way (inaudible)and for me synthesizing is not a process that ok I’m quite and i kinda do it there and i produce know it is ongoing bases so like you know when you’re kinda done with if i can solve a problem then i go for a walk you know you go for a walk  and you kinda think about it in the back of your mind and

speaker  3: oh so like like you go for a walk around town here?

speaker 4: yeah i just go all around 

speaker 3: how often do you do that?

speaker 4: um uh not as often like basically uh depends like sometimes i do it every single day and sometimes every two weeks

speaker 3: its a great pattern disrupt right?

speaker 4: yeah you just kinda interrupt yourself you’re just like okay i uh can’t think to this problem so lets just try something different 

speaker 3: yeah yeah okay what do you look for in a work environment today like vs of what it was when you first started?

speaker 4: um cool technologies technologies is the high one work on right and i guess again uh thats the probably not what a lot of other people will say cause a lot of people will say oh i just you know i wouldn’t be technology egoistic and I’m gonna kinda lured and whatever no uh i think its good to have an opinion on this if i like certain languages then I’m gonna find an environment thats gonna kinda be uh are you using that languages then there is less stuff to kinda debate about and the other things obviously colleges you want other smart people around you you wanna learn from them and you want people you get along with you spend so much time in your life at work

speaker 3: oh man this is so true isn’t it 

speaker 4: yes this is very true so you spend all that time at work you might as well enjoy it

speaker 3: nice we are back we are back somethings is going down on this labor day weekend there is no one in the office (laughs) its late quirtified (1:06:49) theres no one around there is no one in san Francisco this place is a ghost town so lets talk about the good stuff tell me what is the dark side of coding and how do engineers get stuck into it 

speaker 4: the dark side of coding 

speaker 3: yeah the dark side but there is one 

speaker 4: of course theres is a dark side engineering and software engineering umm is uh i guess a lot of people talk about it this is uh there is always constant change so you really have to kinda keep up with that the new technology kinda like whats coming out and that constant change and constant learning can be draining 

speaker 3: a lot of pressure 

speaker 4: absolutely thats why i would not recommend engineering or software engineering to people who don’t enjoy you have to actually enjoy if you don’t enjoy coding if you don’t enjoy learning i would not recommend it because you’re not gonna last period 

speaker 3:how does that impact like learning aspect how does that impact an older engineer 

speaker 4: older engineer 

speaker 3: i mean do you see that its more difficult to learn or theres one that to not learn

speaker 4: i see that sometimes with the older engineers yes i have worked with them there so I’ve seen both I’ve seen older engineers who keep on going you know they just have this energy that just kinda keep on going which is fine you know which is great i see a lot of optimism in there but i see negative energy from the older engineer i guess that there just tried after years and years of it umm and I can appreciate that i mean you get tried and all that but i don’t know i think you have to understand this what I’ve told guys who are older than me and one day methal older than me i guess but uh i told older engineers look what I’ve tried to do with my job is automate my job for years its not gonna happen if i can’t automate my self my job its very impossible for any bit automate themselves that i job don’t worry my job is a security  if you were my job is a security the only thing you end is no job security so you just have to kinda  keep on solving proms and keep on learning and kinda enjoy that and the thing is if you try and take on that mind set and that attitude the other engineers around you are  especially the younger guys especially the guys that wanna kind of learn of you you have so much knowledge that you don’t know about lot of what i do is i learn about all these little things then i argue i don’t ten five percent five percent of the action that i do 

speaker 3: tell me what something you learned is now useless not useless but like its just not practical now 

speaker 4: at the beginning of my career i would’ve poor item i don’t use that or i use it a little bit but i don’t use the object definitely i don’t know there is just i just don’t really apply its not bad to learn these things it expands your mind its like i go i don’t just spend 24/7 coding i go learn about other things i go learn about other subjects health i go learn about uh i read books i read motivational books i read a lot different things 

speaker 3: which ones

speaker 4:theres a lot out growth mind sets a good book never eat lonely is a good book how do win friends influence people is a good book uh seven effective habits of people thats a really good book all these things are good books you take them for what its worth they are motivational pieces and they kinda read them there right and then you read the technical books two i think it’s all good ways to expand your horizons if you’re not kinda doing that and you’re just kinda narrowly focused you have kinda tunnel vision you don’t have none vision around you uh you are kinda get in to this trance where every single day you kinda come to work and you kinda come do the same thing over and over and then time files and you look back and you’re like oh what did i do and like I’m actually in the exact same spot so i really do believe you kinda have to push your self and then in multiple directions and then you know you go back and this and take a step back at what you’ve done and try ti refocus your self again its very important i make a lot of list i uh have usually two or three things i wanna focus on this is what i wanna do but then i allow my self once and while to browse it in it you need that time 

speaker 3: how do you deal with distraction

speaker 4: so distractions again i make list i take what is bothering me and then i kinda write down and i know thats not on my mind anymore so thats what kinda helps ummm a lot of things and a lot of distraction are not important they’re just not and you have to actually learn to say to yourself no those things they’re just not important 

speaker 3: so hard to do isn’t it 

speaker 4: its extremely hard to do i think kinda hard to do I’ve seen it on both sides as an engineer its hard to do because you want to go solve that problem for men its hard to do because you wanna please your bosses or whoever your stake holder is right but its the a lot of times its the right thing to do you have to know that for every signal thing you do takes away from doing what might be important right so you want to kinda of i used ro kinda categorize my list but no i just kinda have one big list and eventually i kinda review it and then i go from top and down you gotta keep the system simple that works for me I’m not saying it works for everybody but for me this having list and inventorying once in a while then going back and focusing on two or main things and then once I’m ready once I’m in multitask mode I’m gonna go for a walk and imi gonna listen to whatever podcast and not do anything on my mind then i come back and kinda leave where i left off and that really helps me with how I’ve been with my mental processing i guess 

speaker 3: and you didn’t orchestra that list into like the beginning part of the day when you’re like bashing things together like is that bad 

speaker 4: kinda  its not the beginning part of my day but the days where i need to focus or the part of my day i need to focus and for me i try to do the multitasking stuff in the beginning of the day i know thats a wash then the focus part of my day and this is different for everybody i just talked to someone this morning her most productive part if the day is in the morning when she wakes up her mind is crystal clear everything she needs to do 

speaker 3: i am totally like that 

speaker 4: yeah yeah and most people are like that too me because my schedule uh my kids i pick them at the school or i you know i handle all that thats actually the distracting part of my day so i only its distracting so I’m gonna plan all my distraction activities around them and it changes if they have summer then i probably gonna do it differently 

speaker 3: what kind of music do you listen to?

speaker 4: pop music i have 3 daughters and six sister in laws and kinda like pop music 

speaker 3: but you listen to this like passively because they request or you actually push that button 

speaker 4: its pretty addictive so i push that button kinda listen to it 

speaker 3: who 

speaker 4: uhh justin bieber mandy morre or i don’t know just pop music yeah i don’t listen to like classical music it relaxes my mind i don’t know that stuff is catchy i don’t know its embarrassing i guess but i don’t really care 

speaker 3: all good how about when you’re working what do you listen to 

speaker 4: you know i find that i don’t listen to too much music while I’m coding i know a lot of coders stereotypical um picture of a coder you put your headphones on 

speaker 3: like in those 90s movies

speaker 4: yeah 90s movies what the most ridiculous movie 

speaker 3: hacker (laughs)

speaker 4: hackers hugh haffermen wolverine right like apparently coding is like putting hexie diagrams and 3d images no thats not how it works like that not how it works, thinking logically.

speaker 4: hackers hugh haffermen wolverine right like apparently coding is like putting hexie diagrams and 3d images no thats not how it works like that not how it works, thinking logically. 

speaker 3: really bad electronic music in the background.

speaker 4: terrible..terrible. and for me thats distracting, i have friends who do that, i have a friend who does exactly that on music, the main thing he is trying to do is focus and remove any distractions. Now if you have a noisy environment i think putting headphones on actually is gonna give you more focus, but if you have a quiet environment around you, why add noise? makes no sense to me. Right?

speaker 3: true

speaker 4: I think it depends on the environment, i think he works in a different kind of an environment where its not as quiet, he does it. I have done it with headphones on when its noisy. I kinda do what makes in that context.

speaker 3: What is the environment around you? What environment do you create around you to get the most done? How does it look? is it in your home? is it in your office? is it like in a restaurant? is it like outside?

speaker 4: I think office environment has some benefits and home environment has some benefits. Office environment i am not around my kids. The wish is great, i love my kids but you know like dad dad, oh great here we go again. At home around there, i have to lock the door and i have to tell wife where i have to focus, there is no other way around it, once you get off that focus track you are gonna have, studies have shown many times, 15 20mins back to gain focus. Its just not worth it. I would rather be very efficient with that time and then get off that time and play with kids.

speaker 3: What time is that when you go to play when you are with your kids?

speaker 4: I think it’s good to kinda structure it and have it kind of regiment it but that kinda need right? yeah usually 5-6 i have like full afternoon basically after lunch till full afternoon and then 6 o clock dinner and hang out with my kids for a little bit and thats it. 

speaker 3: If you had to decide it all over again to the day that you decided to be a software engineer. What would you do differently?

speaker 4: okay i think thats a good question. Finding some guy who has been there before and just have some conversations. Basically, a mentor..mentorship is really important. Somebody who has done it before and kinda have that conversation with him but somebody.

speaker 3: Back to the hot wheels site, is it still around?

speaker 4: no no, a disaster. 

speaker 3: you made some money from it?

speaker 4: like i said i broke even, no money from it. 

speaker 3: it got you a job, so i wouldn’t say a total loss?

speaker 4: oh yeah. not a total loss in that way but yeah ofcourse. I learned alot

speaker 3: so if you were to go back in those days, did you have a like mentor or some kind of example that you copy pasted from?

speaker 4: Yeah i mean, I kinda did my own research and did not have a mentor at the time and i think thats very important.

speaker 3: its pretty simple

speaker 4: i think its important to, i think an engineer focuses on building and you are gonna build something and that doesnt matter. Its like a if a tree falls in a forest and nobody’s around the area the tree never fell.

speaker 3: so true..so if you were to look for mentor, would be things that you wanna gain from them? 

speaker 4: i guess, thats kinda what we were talking about earlier. what we should gain is the skill set thats out of your own comfort zone or skill sets thats in a different discipline. find somebody that you are not good at like amrketing, business side, management side, whatever that you are not good at. Thats what you need..you need to know how to do certain things well. okay? you do that. 

speaker 3: what are the habits or routines that you implement in your day to day work?

speaker 4: I think its good to get on a regiment schedule thats again for me..I am not tryna say but it is as simple as that. allow yourself some leevey like once a week or so to go down and explore you need that time you need toupdated and learning new things. 

speaker 3: when you say explore what do you mean? going to meetups?

speaker 4: yea sometimes going to meetups, casually browsing, reading, let your mind drift

speaker 3: when you do that, where do you usually end up?

speaker 4: still browsing about technology most of the time.

speaker 3: What is the first hour of your day look like?

speaker 4: so more recently my schedule is really different, now i am sleeping from 10pm which is completely different from how i used to beheve before. I wake up at 5am, I start working really, I turn on the light, i brush my teeth and i start working then and there. 

speaker 3: As soon as you wake up, you go to work?

speaker 4: on my computer, yeah

speaker 3: breakfast, kids?

speaker 4: I start making breakfast at around 6, i drink some juice at 6, at 6 30 i start making breakfast, getting ready. 

speaker 3: between 5 and 6, its probably very quite in the house right?

speaker 4: i do what i wrote the night before, so all of my tasks if i have enough time. this is why i think my most important part of the day is afterrnoon, a larger chunk of time. 

speaker 3: you wake up at 5 o clock, and you hit the computer right? you work and pump out some of the most important things you wanna get done and at 6 30, what do you have for breakfast?

speaker 4: I have been doing alot of juicing lately, so i have been getting juice

speaker 3: What kind of juice?

speaker 4: so its juice i make myself, make for the week. Green juices, celery, carrots, kale and al that kinda stuff

speaker 3: sounds delicious

speaker 4: yeah, i am doing it for the nutrition. 

speaker 3: what kind of blender is that?

speaker 4: I have something called omega 8006, its a freaking expensive blender. 

 

speaker 4: and then last thing is frozen  mance it taste like a dessert, it taste like a chocolate dessert, its amazing.

Cameron Peron: It sounds really good

Tung Nguyen: Yeah… 

Cameron Peron: Alright, Nice. What is important to you outside of coding and tech?

Speaker: Ah.. Ok! I guess stereotypical answer like my family, kids there, I have been teaching my daughter, she is just starting kindergarten and we have been teaching her because I started to fall the wagon. With my kids, I went with them at pre-school and we actually did co-op program. The difference between co-op program and regular pre-school which I learnt this along the way is co-op is much more involve as a parent you actually have to be there once a week and its kind of like supervise your kids and all that, you kind of do as part of the community and family.

Cameron Peron: For full day?

Tung Nguyen: For half a day. but its a big commitment like forcefully with my job that i could do this work, if you could do it and spend the time with it can be awesome not every parent can do I understand but this co-op was interesting because the director of the program, every family there that is involved is given a task that is  something to do a job and our job was, the director told me Thom your responsibility will be  is to see the facilities here, your responsibility is to making sure that garbage cans are going to filled up and taken out that garbage cans are placed, laundry is kind of clean, floors are going to mopping, oh wait a minute, am I a janitor? and she is like we dont use that word sanitation engineer. [laugh]

Cameron Peron: VP Sanitation engineer 

Tung Nguyen: exactly and i loved this, right!  and benefit out of this you know you can spend time with your kids and you see them grow and you also learn  how they are kind of family members and how parents raise their kids. that pretty big and so thats what i do in my spare time there. when i say kindergarten i am teaching them how to read and i carry these note cards are here so i can teach her there. one more thing, going back on my background when I was in second grade, I dont really know that I knew ABC..it was that bad the education system I transferred over to. I remember now very distinctively I learnt how to read through memorisation. like how you learn through Chinese language this is make no sense at all. English language you learn through phonetically so I am just gonna ok… I am going to teach my daughter how to read phonetically like her dad, so this is much easier way to learn to read

Cameron Peron: Which one is it memorisation or phonetics.

Tung Nguyen: phonetics, by far, it makes no sense to read English as if this was a Chinese language

Cameron Peron: You learnt Chinese?

Tung Nguyen: I didnt learn Chinese though, but I learnt English as it was Chinese, and i kind of unlearn that over the years

Cameron Peron: you learnt English like you just memorise like the words?

Tung Nguyen: Yeah, Site (34:48) words, studies have shown that the Chao its like a quick and easy way to learn and it seems like it benefits immediately but as the complexity is increased you dont have the fundamental tool to kind of learn 

Cameron Peron: Yeah its crazy 

Tung Nguyen: yeah it is very important to learn the phonetic way 

Cameron Peron: Your background is Vietimi 

Cameron Peron: Why did you learn by memorisation of wars?

Tung Nguyen: Again my parents were immigrants and they really didnt have time to kind of teach their kind of busy, kind of trying to get food on table 

Cameron Peron: so you had to yourself to do memorising words.. right?

Tung Nguyen: Yeah and thats the easiest way to kind of do it

Cameron Peron: Wow

Tung Nguyen: Yeah, you need some instruction for that, yeah thats some of the things i do it in my free time.

Cameron Peron: do you workout?

Tung Nguyen: I used to workout a lot but ever since college, ever since (inaudible) I didnt do that. Shameful to say I should have work out but i think I am going to do a small step like I said what i am doing as I am walking a lot, bike 

Cameron Peron: Yeah its a great start!

Tung Nguyen: I mean thats a good start. I do think that if you put time and energy in that it will worth you with rewards, you just have to commit to it.

Cameron Peron: very good… between you personal achievement if [Inaudible] personal interest how do you priorotise that?

Tung Nguyen: Yeah thats a hard one right. so most of the time people.. I had a conversation with my brother about this. So my brother is a officer and he was telling me that the other officers were telling him dont work OT, thats how officers get extra pay, they just work OT is like all the older officers are like if you work a lot of OT then whats going to happen is you are not going to stay with your kids and then I end up in regret so dont work OT, spend the time with your kids instead.. right! so my brother come simply aware thats very good I aware. so regardless of what you choose as long as you are aware of what you are doing I think thats very important. so me cognisant where  of when i am not spending time with my kids so I kind of separate that out and even though I go to work I feel bad that I am not spending as much time as I am with my kids I make that decision cognitively so I just go ok I know that this week I will spend little time with my kids and I am actually ok with some of that. right but politically correct answer is oh no you are supposed to spend time with your kids. some of the time I spend more time with work then my kids but then sometimes I always like I did the exact opposite and go all in like my personality is very extreme 

Tung Nguyen: Why do you think it is politically correct to say that?

Cameron Peron: to say that, because who is arguing that? all good you are spending time with your kids right versus

Tung Nguyen: I mean your work can argue that?

Spekaer 3: Yeah you can say working right but its..

Tung Nguyen: oh like [inaudible] to work can argue with that

Cameron Peron: let say you are having conversation with somebody else right and then that person is listening to you and you say oh i spent time with my kids like oh yeah…you know.. then they are going to this kind of a rhythmic nodding of their head and there is like of course.. yeah yeah … thats great you know and all that..like its..thats cooperative kind of nice thing to say in normal social context. yes I actually said … no what I am going to do is actually i use that as a reason too work because i think i can get more benefit out of working right now and i think i can with the benefits later right… you gonna manage your time and there is no right or wrong answer here. everybody is different, there are times when I basically say I shutting this part of right now and  work on my thing, right… thats my I go lock the door when i need to and there are times when on a opposite side my no.. i feel like now i need to spend sometime with my kids and lock of work but i try to. it almost sound like I am separating work from personal but i dont do that. and doesn’t work for me, separating personal work, personal life and work life thats not work for me I have tried that.  

Tung Nguyen: (inaudible) same for you?

Cameron Peron: yes, I blended two [inaudible] completely, thats work better for me. like when I am with my wife I got talk about my work stuff and when i am with my workplace I talk to them about my personal stuff like there is no other way to do it. 

Tung Nguyen: It more natural

Cameron Peron: It way more natural. Yeah I think again [inaudible] politically correct way and approach is you are supposed to separate those two from each other I dont do that, i dont do that at all. Why? because I am all in. it makes no sense to me.

Tung Nguyen: so true how do you negotiate or build the right kind of [inaudible] for your wife or your family, its like no sometimes you are going to be too busy and you are not going to be able to do [inaudible] spend time with them, how do you do that?

Cameron Peron:This is hard! its interesting, I explain to her different people how they operate also, I will tell you a story, Richard Bradson another motivational [inaudible]. he talks about an autobiography or biography somebody talk about him, they talk about some of the things he devised grown up and how he was just very determined and he tried to learn swim and he did it for all summer so he couldnt learn to swim at the end of summer wasnt able to swim but then he saw stream while they were driving home and then he just called his dad to pull the car so he jumped over the stream and there he kicked as much and he was almost drowned and he eventually learn to swim so very motivated person. but you know how far did this motivation go. then another chapter in the book talks about when he was racing across Atlantic ocean and trying to beat the world’s record time racing across ocean. for me, when I heard this story I was like ok cool… but then… there was kind of footnote in that story and in that footnote was when he was racing across ocean and then he called his friend up because his engine is broken and his friend helicoptering and do engine on the spot and then they beat the world record time and he called his wife just had his baby and I was like how can say that as a footnote, his wife just had a baby he probably missed the birthing of his child. So I was like you know i think [inaudible] how I negotiate with my wife like look, I am pretty German, I focus and I block out my things at times but look I am never going to be like that. this is why I just decided.

Tung Nguyen: this is amazing how achievement in life, fulfilment with your family,  how easy it can be to twissels around, right?

Cameron Peron:Of course!

Tung Nguyen: or if you are achieving so much, like him, how easy or may be not easy but how the satisfaction with the family can become less important? [inaudible] even know…

Cameron Peron: I think once you can reach that threshold or you reach that level you get away with that I guess more but I guess he is ok with that but that doesnt change my opinion, I am not personally ok with that. By the way I think first of all he is great, he is inspiring, I think its great that he is with virgin group for forty years now thats what why he is resume is amazing

Tung Nguyen: its crazy, you guys have achieved so much, but do you agreee that a footnote if you are building a family there is, at the times of change, right? the way we look at responsibility is having like children stuff is different than without parents right? 

speaker 3: Yes, Of course 

Tung Nguyen: and there is more [inaudible] between like you’re working in personal life for sure, so I would also agree that like footnote .. i dont know part of it is little concerning 

Cameron Peron: Yeah read the book, [inaudiblle] 

Tung Nguyen:  you are the only male in your family, right? is that true?

Cameron Peron: this is true, 

Tung Nguyen: What is this like?

Cameron Peron: I have a brother on my side but I have three daughters and six sisters, so i am the only male typically in our gathering somehow I side here. its ok, there is no big deal. I do my own thing and they kind of girl do their own thing and talk about their business, their gossip whatever. I kind of give m sometime to do whatever i want than. in this situation, I am only male who wear pants thats not true at all. i have like eleven female pants wearing counterparts, right! so all of those other people wearing the pants right? and its ok. I enjoy. I take some enjoyment in a sense that o I have bunch of a females around me and I am cool with that. I dont know there is not really enough words to describe it.

Tung Nguyen: What you wish you would know about coding twenty years ago?

Cameron Peron: Twenty years ago I didnt think it was existed but I wanna answer. I would say when I first started coding. I wish I knew that there are other people out there that [inaudible] dont know a lot of the answers. that clearly dont know the answers and its ok that you just go in there and learn and not necessarily take this experts of ice, right!. Coding is something that again constantly changes and there is a lot of people kind of out there, you see there is wide [inaudible 1:45:03] they go in there and just so super [inaudible] anybody think and then you kind of [inaudible 1:45:12] you are not going to be  able to go out there as a coder and kind of get reassuarnce, a kind of right thing to do, unless you go out there and you do yourself and get that experience and i know that this contradicts over what I say about mentorship, right! but you really have to just go out there and just kind of go for it and not be afraid of failure. i think thats the thing of going back if I just kept on jumping into even more fearlessly than I would have what I have even faster. 

Tung Nguyen: How important is failure [inaudible] for engineering? 

Cameron Peron: Extremely, its something that you are going to fail in software engineering and you’re going to have to learn how to manage that failure. you are not going to write coding right the first time. I just dont see that happening. My coding approach is I get an idea of what I am going to write, and then I start writing it little bit and then I start writing actually what I called black box testing everywhere, that allows me to then refactor everything i need to do. I dont write spect for every little thing, I dont think thats a great use of my time. So I write enough to get the structure in place. well going back [inaudible] thats how I basically rationalise my thoughts and then I write little bit a code and then i write all the test, black box way and then I basically factor [inaudible] because once I have test in place I have a ticket to go to town the code so I could refactor everything out and then that allows me to kind of eventually write to the original concept and original flow to that code that i wanted to write. so that how I 

Tung Nguyen: so is this a process for you?

Cameron Peron: yeah, it because, the test are run so quickly, much more quickly then you normally test again. there is no way you can test it manually over and over again. so of course like everybody talks about like this you have to have testing your code if you dont then I find the fastest coders, the guys who have not all the test but have the right test in place to allow them to explore their code and then rewrite it what they need to. Thats what makes it scalable because you have that button a clip to quickly test all the different logic pass that you were trying to figure out along the way. you just press the button and you are kind of covered. so thats what i do over and over and over and i find many conversation with many developers. a lot of developers like they test every level i just dont think thats necessary worth it and i think other conversation that developers are like ok lets not writing test as because you know really worth it. No no no, you got a right kind of right level of testing to make yourself more efficient as a programmer. thats what i mean reason for itest. it makes me more efficient as a programer no other reason.

Tung Nguyen: Efficient or effective?

Spekaer 3: Efficient and effective I guess you are right. so you want your code to do what you needed to. there is another fact out there that lot of developers say test bad because then you are writing to the spec. and you are not writing what you need. you have to take a step back actually test the product to. test are in there so that you could get another baseline code to ship it. then you go back and see what going to [inaudible] I dont know its just the very innovative process for me not the process like i just sit down, design anything upfront and then its really [inaudible] for me.

Tung Nguyen: When you think of successfully build back in step which companies or projects comes to mind?

Cameron Peron: I think about AirB&B, i think about [inaudible], I think about Guilt. these are values that have great blogs out there, talk about that they do a lot and you know some of this you can tell is marketing, some of the ideas i incorporated back and you just got to fall along the industry trying to you know ok I seem to know what they are doing. I think [inaudible] recently is pretty good company, they fall in and they do stuff in the infrastructure space and so those are some companies I think about. I think about Netflix too

Cameron Peron: In some cases when i see the stuff, like how can you really apply this?

Tung Nguyen: they are very large compnies

Cameron Peron: Where do you see servers architecture in containers going?

Tung Nguyen: i think containers technology is around here to stay, alot of companies are continue to adopt because its a standard deployment mechanism so it makes kinda opertation site kinda easier all the vendors and partners around are gonna start building to it. this is not new. before doctor there was elixis container, google has been using it forever, thats their secret sauces. I am still looking for enough frameworks out there helping with that. 

Cameron Peron: Alright few rapid fire questions for you

Quality vs quantity time with your family, which is the best for you?

Tung Nguyen: Quality, endless time, quantity is just waste of time you just have to be engaged

Cameron Peron: 2nd dumbest thing you have done, not the first but the second

Tung Nguyen: Umm..lets see once i got into this juicing thing right, basically threw everything green in ther and that was stupid. I get in extreme moods. I am extreme person, so once i get to something i like all in

Cameron Peron: is the bay area are full of conventional thinkers?

Tung Nguyen: depends of which part of the area..some of them depeneding on the bay area. if its south bay..more conventional thinkers. people are way more structured…they work for bigger companies. They hire people who are more structured..atleast my friends there are more structured..smaller start up companies are free flowing

Cameron Peron: would you apply conventional thinking to companies that are not high up not big companies like apple, google facebook?

Tung Nguyen: To a certain extent there are diamonds in a rough within those companies. I am not saying thats all you do. Bigger companies you are a peck in the machine.

Cameron Peron: When you are software engineers, what do you talk about moer that you wont discuss with a non software engineer? How does the discussion look?

Tung Nguyen: Now that i am a software engineer why would i talk to the non software engineers?

Cameron Peron: like when you are engaging with other engineers, what do you guys talk about, what do you guys talk abobut, someone who has no development experience

Tung Nguyen: oh i see, software engineers i am talking about software things, with non with i enjoy talking to both, software engineers i talk about trending technologies and everything what they are working on, what i like aabout testing what i dont like about testing. Non software people i dont gtalk about software, i talk about other areas of my life, whats trending in the news, thats how it goes, nobody is gonna be interested in me talking about technology when they are not in software engineer field. 

Cameron Peron: cool, alright, anything else you want to share, you want to discuss?

Tung Nguyen: No i think, anybody kinda starting out, it is very indimidating and scary at the very beginning but i will say straigh out like i did not know much coming into this. Its not that difficult..in the begining what you should do is you should look for companies thats gonna give you more hats to wear that way you get a varierty of different expereiences and you get to kinda understand where you fit in. If you kinda work for a big company, which is really sexy for somebody coming out of college and looking for a job. oh yeah i get to work for this.

Cameron Peron: it is better to get an opportunity where you can get more variety? and choose what best for you then choosing big logos, somewhere you may reapp the consequences in future

Tung Nguyen: thats coming from me and thats my own experience with it but when you kinda get older you will start specilaizing and thats fine too. I have friends who ask me shall i work in a big company or a small company? most of the time my answer is big company, its the opposite answer as they get older in their career. Its just like a business or restaurant owner would tell his son if they ask dad what should i do. They wont say dont start a restaurant, at the beggining of the career that makes a lot of sense.

Cameron Peron: yes it does. Thanks so much for your time. It was wonderful. Enjoyed it.

Tung Nguyen: My pleasure

Cameron Peron: Alright thanks