“We want to reduce friction between teams so that they can move fast, innovate, and deliver on their mission.” – Lukas Sliwka, CTO @ Grindr
About Grindr
- 3.2 Million Daily Active Users
- Redis throughput at 500,000 ops / second
- Services used: Redis, AWS Aurora, Erlang, Redis, Elastic, and more
- 9.999 uptime
About Lukas Sliwka
- CTO at Grindr
- Previously built the technology platform behind Beach Body
- Avid motorcyclist
- Lives in Los Angeles (previously Poland and Canada)
We met with Lukas Sliwka, CTO at Grindr, in their West Hollywood office to learn more about what it takes from both the team and from his leadership to scale up a backed stack to meet 3.2 million DAUs (Daily Active Users). We covered many things, including:
How to scale the Grindr stack from 800,000 and to millions of DAUs and how a near crisis was handled over a holiday in a remote Starbucks parking lot. We also learned how the combination of performance and low maintenance was needed to keep infrastructure in place that led Grindr’s engineering team to consolidate their databases to Amazon Aurora.
On failure, we learned about Lukas’s thesis on failing fast, but almost as important, trusting the executive team to carry their message forward and ensure that the entire engineering team can fail fast and improve. We also learned lessons from a “big bang rewrite” decision, including its impact on new functionality and market competitiveness.
We looked at the evolution of Lukas’s leadership and how a management thesis that blends operations, culture and technology changes over time. In the beginning of Lukas’s career at Grindr, he focused on technology (i.e. still writing code) and as he matured, he focused more on operations (i.e. what processes and metrics). As the team grew, he shifted his attention onto culture (i.e. pushing the button on every single technical hire for culture).
On setting out for success (or failure), we learned what it takes to be successful at Grindr: curiosity and willingness to experiment and try new things. Looking for this trait has allowed both new and veteran talent to adopt quickly.
On goal setting, we learned that a combination of practical planning and brute force has been a proven methodology for Lukas. For Lukas, goal setting starts the evening before by setting a few key objectives. Brute force is not natural but a necessity in order to execute what needs to get done. We also learned how hitting the gym at 5:15AM and waiting to check emails until after the workout sets a practical work/life balance for Lukas.
On luck, we spoke about how to stack the deck in your favor, particularly by investing in the right communities, such as meetups, and building the right relationships with interests that genuinely resonate.
On balancing life interest, such as motorcycling and relationships, Lukas reflects on mistakes he made early on by not putting a priority on delegation.
We also covered how having a well-defined outcome helps to call it a quits or a success. Squishy targets prevents making the proper risk assessments and analysis.
You can follow and reach out to Lukas here:
- Twitter – @lukasrepublic
More resources:
Show notes:
- Decision on why Lukas chose to take on the technology leadership challenge at Grindr 2:00
- How the e-commerce boom and crash of the early 2000s impacted Lukas’s life 3:00
- What the stack looked like at 800,000 daily active users when Lukas first joined 4:00
- Walking into a monolithic mix of custom Java and Ruby on Rails 4:30
- How the Grindr tech team solved a complex geo spatial algorithm written in Java, spanning over 150 Amazon Web Service instances with a migration to more stable 24 instances deployment over MongoDB 5:00
- How to replatform a system by utilizing new technology but still enable a stable environment for the massive organic growth that Grindr experienced 7:00
- How Lukas and his team brought chat back up on Christmas Eve in a Starbucks parking lot 8:00
- Using Erlang to build a stable and high performance chat stack 11:00
- Deep dive into the Grindr stacked: restful microservices and aka based framework 11:00
- Cloudflare for dynamic and static acceleration, including “Argo” for smart routing at low latency 11:30
- Redis Labs to meet a cache running 50,000 ops per second 13:00
- Elastic Search and Aurora for performance 14:00
- Real time chat powered by xmpp and MongooseIM (an IM chat server provided by Erlang for high throughput and high concurrency) 14:30
- Deep dive into how and why Grindr structured their Redis Cluster into 12 smaller databases for better tuning between high read – low write and high read – high write 15:00
- Move chat appliance from dedicated hardware on Rackspace to 200 in production instances over Amazon Web Services 16:00
- Making the decision to consolidate database usage to Amazon Aurora from a mix of DynamoDB, MongoDB, and RDS 18:30
- Elastic Search for geo spatial queries and Redis a write-through cache for all services 19:00
- Predictable vs. stable usage that Grindr faces 20:30
- Using mix of Nginx software-based load balancing in front of key services to manage burst traffic 21:00
- Scaling the team from 4 full-time / 2 part-time members to 90 people that are geographically distributed into 9 full stack delivery teams 22:00
- “We want to reduce friction between teams so that they can move fast, innovate, and deliver on their mission” 23:00
- Building a stack that can handle major international security challenges 24:00
- How Grindr’s “red team” attempts to hack Grindr everyday to elevate stance to reduce attack surface 25:00
- “Failure is not punishment.” 27:00
- How teams are designed to fail fast and grow stronger. This goes so much further than engineering; it must be carried over by the executive team 28:00
- Learning from failure: setting the right time expectations for replatforming the client for better security and operational structure 29:00
- Lessons learned from Big Bang rewrite vs. incremental refactoring on the platform 31:00
- How bad apples can destroy an engineering team’s morale 32:00
- Any technology leader must focus on operations, culture, and technology 32:00
- How the wrong cultural fit can bring down an engineering team 34:00
- The no “asshole policy” and how to deal with damage 35:00
- How curiosity and a sense of egolessness make for engineering success 36:30
- First hour of the day: up at 5:15 and hitting the gym to clear the head 37:00
- Dealing with discouragement and distraction: brute force and goal setting 38:00
- Lessons learned on how to achieve work/life balance 42:00
- How trusting the team enables leaders to delegate 42:30
- Planning the day in 30-minute increments and time for catching up with the family and other relationships 44:00
- How the passion of motorcycling started off as a traffic hack 45:00
- Steering luck so that the cards are stacked to your advantage 47:00
- Pursuing a successful career in the LA tech scene 48:00
- Understanding where your contribution in the startup lifecycle makes the most sense 51:00
- When it’s time to call it quits 52:00
- Making a geographically diverse team work 56:00
- What the future looks like to grow from 3M to 10M DAU 58:00
Full Transcript
Lukas Sliwka
00:00:02.19
You know what, at the time, we were doing probably around just about a million of daily active users, maybe 800k when I joined. I don’t remember exactly the metrics, but it was around there. And the stack at the time was– it was this hodgepodge of custom java development, mishmash with Ruby on Rails services. Very monolithic. And Joel calls me and he’s like, hey, I think chat is down. So imagine me, I think it was Christmas eve or something like that and I’m in a parking lot in Big Bear — Starbucks and I’m literally going through our repository and looking who’s made commits to our chat core and I’m looking these names online, googling them to try to find phone numbers and I’m calling these people to try to get some help. I feel any lead or manager, director, VP needs to focus on the nuts operations that’s culture and technology. I have a very firm no asshole policy. That toxicity needs to be watched for and it needs to be managed very quickly because it doesn’t take long for just even one person on that team to be so damaging that you start losing people. I would say that you can make your own luck. You could say, hey, you’re lucky that I’m working at this company or I’m lucky because I got this job or I’m lucky because of something happened. I think that having an understanding how you want to grow your skillset and career is something that everyone should pay attention to – a lot of people don’t and just kind of go from one job to the next – but you can actually steer that.
Cameron Peron
00:02:06.73
Hello and welcome to the Devibe podcast. I’m your host, Cameron Peron. And here are decode engineering experts behind some of the most complex, high performing, and scalable stacks, to find out what technologies they love to use, what challenges they face, and what their real life is like behind the code.
Cameron Peron
00:02:25.53
Alright, Lukas, welcome to the podcast.
Lukas Sliwka
00:02:29.34
How’s it going?
Cameron Peron
00:02:30.19
Everything is good. Everything is good. Grindr. 3.2 million daily active users.
Lukas Sliwka
00:02:36.64
That’s correct.
Cameron Peron
00:02:38.90
Amazing. Let’s start at the top. Why did you choose to work at Grindr?
Lukas Sliwka
00:02:42.30
You know what, when I was approached by Grindr around four years ago, I didn’t know much about the platform or what the app was. And once I started researching very quickly, I realized that this was a platform that was already global, that was serving underserved group of users, and was able to provide a service that allowed them to connect with each other. And that is very interesting in itself, very interesting engineering problem. Doing that on this global scale is another. But also when I started talking to the — Joel Simkai who is the founder of the company, he really wanted to make sure that we started investing in engineering. Grindr at the time had a lot of stability problems, they struggled with being able to push out features very quickly, so the challenge was to not only focus on the right technology to scale globally, but also to build the right engineering team to be able to move quickly and that was the challenge I was actually looking for at the time and I ended up joining the company in October of 2013.
Cameron Peron
00:04:04.44
Ok great. What were the major events that brought you to join Grindr?
Lukas Sliwka
00:04:08.73
Career wise?
Cameron Peron
00:04:11.11
Yeah
Lukas Sliwka
00:04:11.68
I think that…it’s an interesting question. So, when I look at that, I sometimes think what makes someone ready to take any challenge, right? I felt pretty strong about being able to take on this one because when I look at my career there’s been kind of this steady progression. I started in the late 90s just developing at the time first ecommerce systems and trying to take some of these companies to the web and I was kind of down in the trenches during the ecommerce boom and then worked through the crashes and then joined a couple startups. I’ve worked for big companies, small companies, so I’ve kind of seen all the different ways to kind of run engineering and develop software. So by the time Grindr approached me, I had a various different experiences good and bad on different sides of the engineering organization so I felt pretty strongly about being able to set up a good engineering organization and I was also very opinionated about what a good engineering organization looks like. And just so it happened, in conversations with Joel, he also shared that vision and he just needed someone to implement.
Cameron Peron
00:05:44.04
So today Grindr is at 3.4 million
Lukas Sliwka
00:05:52.47
3.2
Cameron Peron
00:05:54.03
3.2. When you joined in 2013, what did the stack look like? What was the low that you were getting?
Lukas Sliwka
00:05:58.50
At the time, we were doing probably around just about a million of daily active users, maybe 800k when I joined. I don’t remember exactly the metrics, but it was around there. And the stack at the time was…it was this hodgepodge of custom java development, mishmash with Ruby on Rails services, very monolithic. Team at the time spent a lot of…I think before I joined, the team spent close to two years trying to build this new stack and unfortunately they invested some time in building things they were already solved for. So I can give you an example. When I joined, there was this geospatial cluster powering the geospatial search and it was this custom implementation of the open source geospatial algorithm written in java that was running extremely hot, I think 150 instances in Amazon and it was very unstable, going down all the time. So one of the things that we’ve done, in the first literally couple of months, we replaced that with mongoDB at the time. So we migrated all the data sets necessary to power that thing to mongoDB and partnered with mongoDB at the time and we went from running this custom thing on 150 instances to running mongo on like 24 instances so there was a big win there cost-wise but also stability and predictability. I’m not a very big fan of building things from scratch when you already have tools either commercial tools or open source tools that are solving for these things. There’s plenty of problems to solve and it’s challenging sometimes when you manage engineers to make sure that they don’t work on this engineering web dream project to kind of implement something that’s very very cool maybe, but that’s already been solved for.
Cameron Peron
00:08:20.54
What are some of the craziest backend development challenges that you’ve faced?
Lukas Sliwka
00:08:24.52
I think it was just the massive and rapid growth of the company. The DAU count, this is mainly organic growth. It’s amazing to me because we have just created a marketing department about a year ago, so regular things like user acquisition and managing that funnel and going out there and actually making sure that you do all the right things in terms of email campaigns and engagement campaigns kind of bring the users to the app and keep them there – that is a recent thing for us. So the growth really happened through the word of mouth and because of the amazing service that the app provides. The biggest challenge for me was how to re-platform a system. How to do it in a way so you invest into these next generation technologies while also keeping the service up. And that’s not easy at this kind of scale.
Cameron Peron
00:09:38.70
Is there any specific event that comes to mind?
Lukas Sliwka
00:09:42.31
I keep talking about it with some of my friends and also developers here. My second month at Grindr – so I joined on Halloween 2013 – and silly me, I planned my Christmas in Big Bear. My mom flew down from Toronto, Canada, my sister flew down from Toronto, Canada, and they enjoyed the sunshine but they said ok let’s see some snow and let’s go to Big Bear. And silly me, I think this is going to be just a nice holiday. And on my way up there, we’re just maybe 30 minutes away from the cabin, I still have cell reception and internet and Joel calls me and he’s like hey, I think chat is down. So I went to the cabin and dropped my family and I didn’t have really good internet. So I grabbed my laptop and I drove down to Big Bear and I started squatting in the Starbucks parking lot and mooching off their internet and I’m like and we’re trying to figure it out. At the time there’s just a handful of guys at the company, couple of consultants somewhere in the basement, literally a couple of guys helping us out and then we had maybe four developers in house. And we’ve been trying to bring the chat up and it just would not come up. So imagine me, I think it was Christmas Eve or something like that and I’m in a parking lot in Big Bear — Starbucks and I’m literally going through our repository and looking who’s made commits to our chat core and I’m looking these names online, googling them to try to find phone numbers and I’m calling these people to try to get some help. And we finally– I was able to finally connect with the guys who actually made some modifications there and they connected me with our current partner, Erlang Solutions, and those are the guys in Poland, they have a group that is very skilled at Erlang and our chat stack is Erlang based so they were able to help us out and bring things up. But those are kind of the worst stories that I would say would happen. Over the first nine months while I was at Grindr it was all about trying to stabilize the patient on the table, stop the bleeding, focus on some key metrics like uptime and time to resolve issues and start slowly introducing these basic engineering processes in place to start growing a good engineering shop.
Cameron Peron
00:12:34.72
What does the stack look like today?
Lukas Sliwka
00:12:36.21
Today we are on a version of stack that is primarily powered by microservices so we’ve got a whole bunch of rests for microservices. They’re exposed through a stack that uses cloudflare as our CDN and we use that for both dynamic and static acceleration of various different endpoints. Also for security, cloudflare is amazing security wise. And then that assists with all the geo routing and we recently implemented something called Argo from cloudflare which is this smart routing, global routing where they constantly detect any issue or congestions on the internet and they are routing your traffic to kind of reduce the latency around the world. And then the stack is powered through various different microservices that are essentially written in java and they sit on top of this ACA based framework. And ACA is Scala based. So all the restful APIs are pretty much ACA framework, the business components are written in java, and we’re leveraging heavily redis, through redis labs, or redis labs another partner of ours. We’re actually now working with them to optimize that caching layer because our cache is pushing around half a million ops per second which is crazy, so we’re doing some optimizations there and then we’re using elasticsearch heavily and we’ve recently migrated pretty much all of our storage to Aurora. We see amazing performance from Amazon Aurora and then on the chat side, the chat side is powered by stateful connection and that is powered by XMPP and the main engine there is MongooseIM which is this erlang based chat server and that is kind of whats powering our real time chat. So you’ve got really two sides of the coin, you’ve got real time chat and that’s erlang stack and that’s high concurrency type of uplines and then restful services.
Cameron Peron
00:15:05.17
How big is the redis cluster?
Lukas Sliwka
00:15:08.41
I don’t know from the top of my head, but last time we were looking at it, it was that, I think it was like this multi master, multi slave deployment with I think around 24 slaves and a couple masters in the cluster and there was a proxy in front of it. And then now we are actually migrating – so we’re splitting that into multiple smaller databases. So the thought is that each microservice should have its own caching database and then have its own storage. So the direction that we’re going to is essentially over the next six months, we’re going to split that massive database into probably 12 smaller ones and each of those then can be tuned better because then you can kind of see the utilization pattern behind each API and say hey this is maybe high read/high write and it’s a very different type of tuning that you need to do or you can do, versus say transactions that are maybe high read and low write.
Cameron Peron
00:16:26.05
It seems like you’re running this on Amazon, is that right?
Lukas Sliwka
00:16:29.39
That’s correct. So we used to have our chat uplines in rockspace on dedicated hardware for a while. One of the problems that I encountered when I started was this chat would go down because of the underlying amnesia database and Amazon at the time, 3-4 years ago, these net splits would happen and amnesia is just not very happy with that and the whole thing would just get into a state where it was unrecoverable. So we moved our chat to dedicated hardware on rackspace. But since then, the stack, I mean the latest upgrades on the amnesia side and MongooseIM made that stack much more fall tolerant to net splits and also Amazon has gotten much much better when it comes to their availability so we’ve recently moved all that stuff into Amazon and now we’ve consolidated and all services are actually running on Amazon.
Cameron Peron
00:17:34.87
How many instances are you currently running on Amazon?
Lukas Sliwka
00:17:39.42
So I think all in total if you were to just kind of count everything, it’s probably 800. But production is probably running on around 200-250 instances of various different types and then everything else – our development environments and QA unload test environments and all the kind of ETL and data science jobs and all that type of stuff.
Cameron Peron
00:18:12.36
What was the decision like to move to Amazon Aurora specifically? It sounds like you were using something else before that and —
Lukas Sliwka
00:18:19.90
Yeah, so we had this hodgepodge of various different stuff. So we had- we were using RDS and we were using Dynamo and we were using MongoDB and there’s been a lot of experimentation about possibly using Cassandra or MariaDB and percona, mysql, we kind of looked at a whole bunch of different stuff and Aurora came out and we kind of put it through the paces, the combination of the performance and also the low maintenance necessary to keep that infrastructure in place made us decide that we should consolidate. So we actually migrated a lot of our databases away from MongoDB and Dynamo and all the different hodgepodge of different storage and we kind of consolidated in Aurora. And in addition to Aurora like I said, we’re using elasticsearch and that’s powering our geo spatial queries and we also use redis as this write through cache for all our services.
Cameron Peron
00:19:37.86
It seems like there’s some unpredictability in the events that can suddenly like throttle up usage, right, like gay pride events, a sudden like event of people meeting together, in any kind of urban center — can you talk about that a little bit and what that looks like on the stack?
Lukas Sliwka
00:19:55.17
Absolutely. So we most of the time we enjoy pretty steady traffic so we are not a necessarily a bursty application. This is not like let’s say stubhub or Ticketmaster where they’ve got some sort of ticket sale going on and they go from zero to here in a couple of seconds, and the infrastructure has to scale. Having said that though, we do have various different events that happen so, like, June is all pride throughout the various different countries, so we do have to prepare for that and we rely on Amazon and the elastic low balancers to kind of take on that load. But having said that, ELBs are not very good at bursty traffic, so we do actually have a mix of engineX software based load balancing in front of some key services that tend to get bursty and that works out much better.
Cameron Peron
00:21:04.34
What does the engineering team look like here?
Lukas Sliwka
00:21:07.30
It’s– Every quarter when you ask me that question, it’s a different answer. So probably the most exciting thing about this place is that we’ve been evolving and growing like crazy. So I mentioned when I started there was a handful of guys here, maybe the whole team in office was four people and maybe 2-3 additional consultants, right? And now we have a team of close to 80-90 people, geographically distributed. And all the team is actually organized now into these full stack delivery teams. And we’ve got essentially 9 delivery teams, each team is kind of like a mini startup within Grindr. They have their own business mission, they have their own business KPIs that they’re driving, and each team is composed in a way so they have all the functional knowledge to be able to execute without being dependent on other teams. So a team could have iOS developers, Android developers, backend, devops, QA, they may have some designers, and they really operate very independently. And the idea is that we want to reduce that friction between the teams and enable them so they can move fast and innovate and deliver on their mission.
Cameron Peron
00:22:36.21
We spoke earlier about some unusual events or external forces that have enabled you to recreate the stack. Specifically about in some countries or some groups that are targeting gay men through Grindr. Can you speak a little bit about that?
Lukas Sliwka
00:22:53.95
Sure. One thing that I realized very quickly when I started working here is that we have a very big target on our back. And people tend to forget that in the US or in Europe, we enjoy a great deal of liberties that we just take for granted. And you don’t have to look far and there are countries where being gay is considered against the law or it’s being shunned. We don’t have to look far – Russia is a good example and then if you go to the Middle East you’ve got Egypt, and there’s a lot of countries also in Africa where being gay can be challenging. So when it comes to Grindr and security and making sure that we provide service while protecting our users, working here takes it to a little bit different level. I can tell you that as part of the replatforming of our stack, we’ve spent a lot of time investing into security on the client side and the mobile, I mean how data is transmitted between the client and the backend. We spend a great deal making sure — we try to hack ourselves so we partner with third parties to do quarterly security audits, but also we’ve created a red team inside of our organization and these guys report directly to me and all they do is try to hack us everyday to elevate our stance and reduce the attack surface. But also we actually took it even further because we are investing in specific features that protect our users in bad neighborhoods. So we call countries with either having legislation or maybe anti-gay sentiment, we call these countries as bad neighborhoods and through geo fencing we’re actually able to detect when our users in those areas and we turn off certain things in the app to protect our users further. There are some pretty interesting things that our teams are working on right now, as a sneak peek, where you may have a password in the app that lets you in but you may have another password that you enter in that does a complete wipe of your Grindr and Grindr data. So imagine someone stops you at the checkpoint and they want you to login to the app, you can enter a password which is kind of under duress and that wipes your Grindr data so you’re protected. Or masking the Grindr app so somehow on the phones it’s not easily found. So those are the type of things that we can always do to help our users and then disable some features that make it easier for people to triangulate or locate someone.
Cameron Peron
00:26:25.69
Great. How important is failure in engineering?
Lukas Sliwka
00:26:31.29
So
Cameron Peron
00:26:33.66
Moving on.
Lukas Sliwka
00:26:34.79
Well, without failure we wouldn’t be where we are today. I think from the very beginning one of the goals that I set for our company was to build a culture where we have emphasis on learning and failure is not punished. So everything that I do today in working with our management team is to make sure that teams are able to fail fast, learn from that, and then grow stronger. That needs to be part of the culture and it cannot only be on engineering. That has to also be recognized by the executive team and my job as a CTO is to carry that message and educate everyone. So by now it’s a big part of our culture to be able to fail fast, learn from it, and then improve.
Cameron Peron
00:27:29.96
Where have you yourself failed as an engineering leader and what have you learned from it?
Lukas Sliwka
00:27:35.32
Wow, all these tough questions.
Cameron Peron
00:27:37.22
It’s the Devibe podcast, man, not a walk in the park.
Lukas Sliwka
00:27:44.43
So there’s been some decisions. Looking back on the last four years there are some decisions where, thinking back, I could’ve done some things differently. So I think the biggest one that’s always on my mind is the decision that we’ve made to replatform our client. So in the first 9 months when I was here we pretty much knew that we needed to replatform the backend and the cloud architecture and all that stuff. And then also we realized that we needed to replatform our clients in order to have a better security core and structure the app better so the teams can actually move faster and independently. So we actually decided to do this big bang approach and at the time we said, okay, we can do it in 6-9 months and I sold it to our founder and he was like okay, how long it’s going to take, we’re just going to hunker down and we’re not going to be delivering any features, you guys need to replatform the client and then we’ll– the promise was that once we’re done the clients are going to be much much better, we’ll have all the code coverage in place and all the different tools in place to be able to move fast. And that project essentially took a year and a half. It was tough. A big bang rewrite like that is always very challenging as the months drag on and you’re bogged down in this effort and you’re not really delivering value to the customers, it’s getting tougher and tougher. So in hindsight I would take a different approach and essentially do that effort with incremental– as an incremental refactoring, which would probably be much harder and it would’ve taken longer, but I think we would’ve gotten to the same spot in terms of the stack in stability and security but at the same time we would’ve been able to also deliver some features to our users. So there was a time of essentially a year and a half where there was no new functionality and from the perspective of being competitive in the market that was a wrong decision. And if I had to do this again, I would probably take a different approach.
Cameron Peron
00:30:32.68
What is the impact of bad apples, like on the team? And how do you deal with that? Bad apples as in situation could be like you have a very talented person but he’s not vibing well either with his teammates or with managers.
Lukas Sliwka
00:30:50.57
So listen, I think as an engineering leader there are really three aspects that I feel any leader, manager, director, VP needs to focus on. And that’s operations, that’s culture, and technology. And there’s a different blend depending on where you are and what you’re doing. You may be doing 80% tech and 10% operations, 10% culture if you’re maybe just a lead your focus is very technical. I can tell you that I kind of evolved through being here. From the very beginning my focus was very technical. I was still writing code, I was still involved in the cloud architecture and actually making stuff happen. And over time as the team started to grow, my role started focusing more on operations which is hey, what processes do we need to have, what kind of metrics, what kind of tools, how are we going to organize ourselves in a way so we can actually do it, right? And then as the team grew, my focus then became more and more on culture. So now, the main function I think– I’m still very much involved in technology and operations. But the reality is that we have a very good team that drives those. I’m kind of a culture watchdog. So I’m very much still involved in hiring, every single person. And people that I look for are these jack of all trades but masters of few, people who are extremely collaborative. Because I’ve seen what happens when you hire someone who is not a good cultural fit. Not a good technical fit is much less damaging than having someone who is not a good cultural fit because on the technology side – that stuff is easy, you can always learn from your peers, you can always start pairing, you can go to some training, or whatever. Cultural aspects are driven by so many different factors, it’s so fuzzy and squishy. But I have a very firm no asshole policy. So I don’t hire and I let go of people who come here and become very abrasive or start controlling information or they’re not collaborative and they’re kind of becoming very toxic to the team. So that toxicity needs to be watched for and it needs to be managed very quickly because it doesn’t take long for just even one person on the team to be so damaging that you start losing people.
Cameron Peron
00:33:49.89
What does it take to be a successful engineer here?
Lukas Sliwka
00:33:55.88
I think number 1 is being open minded and being very curious. I probably value curiosity and willingness to experiment and learn new things more than anything else. Because that sort of thing, that kind of trait allows people to adapt very quickly and if they’re put in an environment that fosters innovation, amazing things start to happen. So successful engineers at Grindr, they definitely have that trait. Another thing I want to say is that no ego. I have no ego. When I see that someone has a big ego, they tend to be an asshole. And my job right now is, to be honest, to watch out for those people in the interview process and make sure they don’t work here.
Cameron Peron
00:35:01.20
What is the first hour in your day look like?
Lukas Sliwka
00:35:05.94
I commute pretty far to work so I try to throw in some gym time so I usually get up early. I’m up by 5:15 and I go to the gym, I’m done by 7, so I wake up, start my day with a glass of water with some lemon juice and hit the gym, clear my head.
Cameron Peron
00:35:29.54
Breakfast?
Lukas Sliwka
00:35:31.35
I would just have a shake after the workout and that’s my breakfast. I try to not check news and emails until I’m done with the gym. Because if I pick up my phone and start looking at the emails and news or whatever, I usually get sucked into stuff and it doesn’t work.
Cameron Peron
00:35:59.56
How do you deal with discouragement and distraction, both personally and also like in your professional life, it goes both ways.
Lukas Sliwka
00:36:08.48
I think my approach sometimes is just brute force. And something that just works for me is I do goal settings and I try to– before I go to bed I actually write out some things I want to accomplish the next day on a piece of paper. So that is my way to keep myself focused. When I said brute force, I would just say hey this is what you need to get done and get it done and just push through.
Cameron Peron
00:36:44.31
Is that a natural thing for you or is it something that you learned?
Lukas Sliwka
00:36:47.70
No, it is not, this is something that I’ve learned from various different people and I constantly look for tools that allow me to operate at a higher level. I love reading books, I’m reading right now Tools of the Titans, by Tim Ferriss. I’ve found that as I progress and as the team grows you just have to keep developing yourself because if you’re static, then the organization or life starts outpacing you and your ability to deal with things, and that’s when things get hairy.
Cameron Peron
00:37:27.07
How do you– it seems like you have a tremendous amount of responsibilities and also the challenge of dealing with fires when they come out. It’s a real issue. How do you find time or how do you balance your own personal interests – we could talk about a few of those – and also your own relationships, outside of the heavy workload that you have?
Lukas Sliwka
00:37:48.45
Haha, so listen, I think that I would say the first two years at Grindr were very difficult. I mean, I can probably say this was like a 12-15 hour commitment for the first two years. And during that time there was really no personal relationships, it was all work and I managed by doing some sailing on the weekends or just riding my motorcycle. But I don’t – in hindsight I never want to repeat something like that.
Cameron Peron
00:38:28.30
What do you mean you don’t want to repeat?
Lukas Sliwka
00:38:29.99
You know, I think I just overcommitted myself at that time. I should have found a way to maybe work less and delegate more and maybe hire better people or more people. At that time I think I was more in a mindset that if I didn’t do it then it’s not going to get done and that is something that I needed to learn — unlearn. Unlearn and just empower your people, empower the team, trust. Develop that trust. Because then that allows you to do other things and kind of look beyond the fog of war. And today things are actually very good. It comes down to stability, right? If you’re fighting fires every day then it’s very difficult to look beyond that. For over a year and a half now we’ve been in a place where our uptime is around 3.9 sometimes 4.9 every month. And even outages that we have, the way that the systems are structured now it’s like it’s never fully down, it’s maybe one microservice has an outage and the app is designed in a way that the functionality just kind of steps down but everything else still works, so to the user it’s less visible. So things have been much better. As far as personal relationships, I don’t know if we should talk about that, ha.
Cameron Peron
00:40:09.93
We don’t need to talk about that, but how do you find the balance. Is there a certain time of day you shut the computer off and you focus on the things that are important to you, whether it’s people or motorcycling, sailing, reading, how do you get into a workflow where you can focus?
Lukas Sliwka
00:40:25.44
Got it. So listen, I compartmentalize my day. So I plan my day in probably 30 minute increments when I’m at work and then after that I also have a set time where I do my, I have my reading time and I have the time scheduled for where I will catch up with my family on the phone or whatever. I tend to use Google tools and everything to help me organize my day and I try to stick to it. Where the wheels come off the bus is when we have something unexpected. So if there’s an outage or something happened or whatever, that’s when the plan takes a backseat and that’s when it gets difficult.
Cameron Peron
00:41:15.83
Let’s talk about motorcycling.
Lukas Sliwka
00:41:18.12
Yeah, let’s talk about motorcycling.
Cameron Peron
00:41:20.25
What is it?
Lukas Sliwka
00:41:21.52
It’s just…I don’t know what it is, I got into it probably five years ago and it was because at the time I was working at Santa Monica and I lived in Long Beach and a friend of mine was like hey you’ve got to get a motorcycle, otherwise you’re going to be sitting in traffic. So I took on the rider’s safety course and I got my first motorcycle and I started riding and I just love it. There’s something about it where you’re just on your motorcycle and you go and there’s that freedom that you have. I actually commute most of the time by motorcycle from Long Beach to West Hollywood which is pretty interesting in rush hour in LA.
Cameron Peron
00:42:11.91
Are you one of these guys that’s weaving through the traffic?
Lukas Sliwka
00:42:14.28
I’m one of those guys that tries not to do it but the reality is that you kind of have to, but I never go balls to the wall between the cars. That’s an accident waiting to happen.
Cameron Peron
00:42:28.28
A few more questions for you.
Lukas Sliwka
00:42:31.74
Sure.
Cameron Peron
00:42:32.57
Being a successful engineer – it falls between a spectrum of like luck and pure skill and expertise. How important is luck in the career of any engineer?
Lukas Sliwka
00:42:44.90
I would say that you can make your own luck. You could say, hey, you’re lucky that I’m working at this company or I’m lucky because I got this job or I’m lucky because of something happened. I think that having an understanding of how you want to grow your skillset and career is something that everyone should pay attention to – a lot of people don’t and just kind of go from one job to the next – but you can actually steer that. You can spend the time to go to meetups and grow that professional network and make the necessary relationships that enable the growth in the future. And that’s important. As an engineer when you’re getting that first job and you’re starting out, over time you have to develop those relationships and ideally you want to know what you want to do. What type of things you’re good at, what type of things excite you and do something that you feel passionate about every day. Otherwise it gets boring.
Cameron Peron
00:43:56.51
Is LA the right place to pursue a career as an engineer?
Lukas Sliwka
00:44:02.25
Absolutely. I think LA is a very interesting spot because you’ve got – we’ve got the Silicon Beach, we also have tech distributed throughout the south bay, and you’ve got a very diverse ecosystem of various different companies. So if you want to work in healthcare, you can, if you want to work in automotive, there’s a lot of automotive going on. Same with insurance. If you want media, there’s a lot of media companies. We also have some hardcore engineering companies here, so you have a wide range of various different companies that you can align yourself with. And it’s not as techy or it’s not as in your face like, say, San Francisco. So it’s maybe a little bit more laid back.
Cameron Peron
00:45:08.65
Is LA a good place if you look at someone who is starting off versus someone who is pretty experienced? Where does it make the most sense?
Lukas Sliwka
00:45:18.64
I don’t think it matters. If you’re starting out here, for the most part I think you have the same options that you would have in the Bay Area, because a lot of those companies are actually here. So if you want to work for Facebook or Netflix or Google and these larger powerhouses, you can. But if you want to work for a startup and start in that area also Silicon Beach has plenty of startups that are happening. So I think it’s more of a question of figuring out what you really want and what it is that you want to learn. I can tell you that in my career, I’ve worked for large large companies, I’ve worked for startups, I’ve worked for small companies, the growth companies, and I know what my sweet spot is and i know what I like to be doing. I enjoy taking a team of ten or 20 and growing that to 150-200 engineers and making that work. I have no desire to lead an engineering or tech company or organization of like thousand or 2000 people. Those are different type of problems that need to be solved. I like being still hands on and down in the trenches.
Cameron Peron
00:46:47.75
Cool. When is it time to say quits? Not like per job, but per project or an initiative that you’re working on where you’re collaborating with people together.
Lukas Sliwka
00:47:02.13
I think it really comes down to making sure that you have a well-defined outcome for everything that you do. When I see a lot of spin, both in personal lives or spin in terms of projects or engineering it’s if you have a very squishy target that tends to cause the spin because you really don’t truly know what you’re after. So making sure that whatever you do, you have a very clear outcome in place and you check against that is key. Because that allows you then to assess and do that risk-reward type of analysis and hey I’ve been banging my head against a wall and we got to a certain point and is the juice still worth the squeeze, so to speak? And without having a clear outcome in mind, it’s difficult.
Cameron Peron
00:48:04.41
What kind of music do you listen to, both at home or when you’re working?
Lukas Sliwka
00:48:10.32
Oh man, I’m equal opportunity kind of a guy so I tend to listen to a lot of different things. There are days where I would have some ADM track on and get off on that. But there are some days where I would just listen to maybe some elevator jazz or whatever, it really depends. It’s kind of funny, there are some people who are very religious about the types of music and I’m kind of equal opportunity there, depends on the mood.
Cameron Peron
00:48:48.64
Are you currently hiring for any open positions?
Lukas Sliwka
00:48:51.92
Oh man, we’ve got I think like 15 positions opened. We’re hiring for our mobile engineering team, we’re hiring in the backend, we’re hiring data engineers and data scientists, we’re hiring audio facilitators and the reality is the team is still growing. We’ve got like I mentioned, we’ve got nine delivery teams, around 80-90 people. Everyone is geographically distributed but 60-70% of my team is either in, foremost in Europe or Argentina or in Mexico. Over the next 2-3 years, I would like to change that ratio and have maybe 60-70% of engineers here in the US. It’s a little bit easier to manage the teams.
Cameron Peron
00:49:47.56
What are the kinds of challenges that you face when you’re managing a team that is distributed like that?
Lukas Sliwka
00:49:51.44
Well, the biggest thing I think is obviously the cultural differences, so being aware of that. I think people underestimate the churn and difficulties that come from not being in the same office. But here’s the thing – you can definitely have very high performing geographically distributed team if you’re doing the right thing. I can tell you one of our highest performing teams is actually located in five different countries. We’ve got guys in Poland and Croatia and Argentina and US and Mexico. And they all come together and they totally kick ass. But they are truly an agile team. They determined together their working hours and they shifted the hours accordingly and they’re very very successful. Having said that though, you’ve got to give them the tools to be able to do the job. So having a good teleconferencing system, making sure that you’ve got constant telepresence and streaming video from different locations to bring people together. Making sure that you have the right tools in place to allow the remote pairing and code reviews, all that stuff, you’ve got to make sure that all these things are in place to reduce the friction.
Cameron Peron
00:51:25.01
Any final notes that you want to say?
Lukas Sliwka
00:51:29.71
Final notes…not really. I think it’s been great. Looking back when I joined Grindr, I definitely did not expect various different turns and twists that happened over the next four years, but I can tell you that as a company we’ve matured a lot and now there’s this next chapter beginning for us where we are driving towards that 10 million DAU mark over the next 2-3 years and that’s going to come from all the marketing efforts and organic growth. And we’re going to be focusing on specific markets, the growth markets for us and maybe doing very specific market studies and start developing features that are targeting just those markets. So this is going to be very exciting for us.
Cameron Peron
00:52:27.21
What’s it going to take in order to grow from 3 to 10? That’s a huge number, a huge leap.
Lukas Sliwka
00:52:33.56
So it’s going to take several different things. As a company right now we are diversifying, so four years ago Grindr was just one app. Now we have multiple apps in the app store and one is our Korab? The other is our Gaymoji which is our play to kind of get into content and multi-app ecosystem. We’ve also launched INTO magazine which is essentially our content play and we are now publishing around 10-12 articles per day which are very high fidelity high value type of content. And we’re going to be going after lifestyle features, so Grindr will be that app that a gay man uses for everything hopefully, in the next 2-3 years. Not only to meet people but also to go to places, discover places, use for travel, maybe ecommerce, content, all these different verticals are going to come together and generate that sort of traffic.
Cameron Peron
00:53:54.10
Nice. What can we expect in the next year?
Lukas Sliwka
00:53:56.87
I think next year there’s going to be more features that are driven by our data science so we’ve been investing heavily in our data science and data innovation team. You will start seeing features that are driven by user behavior, that are driven by geolocation and what’s going on in various different areas of the world. So we’ll be bringing all that stuff together to power these more advanced and smart type of features. And that’s for the purpose of essentially bringing the right services, bringing the right content, bringing the right places to our users based on their preferences and behavior that we want to discover in a passive way. So I don’t want to be pestering people with questionnaires and whatever. Perfect case is like I know that as a user you may be going to sushi restaurants or you may be working out at Equinox as opposed to some other gym, so the very interesting question is how can I use that information to provide better service? And there are some unique opportunities there in terms of a recommendations engine or recommendations for content or travel, it’s going to be pretty cool.
Cameron Peron
00:55:25.27
Nice. Cool. Lukas, CTO of Grindr, thanks so much for joining us today.
Lukas Sliwka
00:55:30.86
Thank you, this has been great.
Cameron Peron
00:55:32.04
In the show notes we’ll put a link to the careers, a few notes on the books as well and music. Cool. Thanks so much. Awesome, take care.
Lukas Sliwka
00:55:46.80
All right, bye.
Cameron Peron
00:55:47.98
Hi, thanks so much for listening. Your feedback is really important to us, we really want to learn how we can make it better. Let us know who you want to listen to, certain questions you want us to ask. Please reach out to us on all the socials. You can find us @devibepodcast on Twitter, Instagram, Facebook. Feel free to email me directly at [email protected]. Thanks again.