Migration Makes My Skin Crawl: From SQL to NoSQL

Listen to the on , , , , or wherever you listen to podcasts. Hacker Noon Podcast Apple Podcasts Spotify Google Podcasts Stitcher Is database migration as scary as it sounds? Amy Tom talks to Matt Groves, Senior Product Marketing Manager at Couchbase, and Curt Gratz, Co-Owner of CKH Consulting. Amy, Matt, and Curt talk about migrating from a SQL database to a NoSQL database, the challenges developers face when shit hits the fan, and assessing the use case of your database. Curt is a Couchbase user and is well-versed in database migration; he shares how to avoid disaster migration and what he's learned using a NoSQL database. Matt slams down the expert advice on what "NoSQL" encompasses, and how to convert your data structures. In this episode, Amy, Matt, and Curt discuss: What considerations are behind the decision between using a SQL database over a NoSQL database? (05:05) When is NoSQL an inappropriate solution? Matt says it depends on the amount of data and your need for adaptation/personalization (06:42) What are the challenges facing developers moving from SQL to NoSQL? (08:48) What the heck is ETL? Curt explains that "ETL-ing data" means transforming your data from one place to another (10:10)Can we automatically convert the contents of a SQL Server database to a NoSQL database? Matt talks about his automatic translation project that lifts and shifts data into Couchbase (11:30) What data structures need to be converted when migrating from SQL to NoSQL? (13:12) If NoSQL is schema-less, do the schemas also migrate over? Long story short, Matt says to think of NoSQL as "schema flexible" instead of "schema-less" (15:57) What's the deal with stored procedures and how does that impact NoSQL database operations? (17:30) What about ACID and atomic operations when migrating to NoSQL? (20:46) What happens when shit hits the fan in the migration process? Can you lose your whole database? Curt talks about disaster migration scenarios and how to solve them (23:07) What about the process of migrating from one NoSQL database to another? (29:05) What is Curt and Matt's advice to first-time migrators? Curt says to ask yourself, "why am I migrating?", and Matt says to expect things to go wrong (30:52) Follow Matt Groves & Curt Gratz: You can follow Matt on Twitter @mgroves You can email Matt at matthew.groves@couchbase.com You can follow Curt on Twitter @gratzc You can email Curt at gratzc@compknowhow.com Shownotes: You can check out Matt's SQL Server To Couchbase on GitHub You can learn more about CKH Consulting: https://ckhconsulting.com Machine-Generated Podcast Transcript (Please Excuse The Errors) I've been hosting podcasts for a little while now. I like to think that we've moved past the acquaintance phase and we're really more on to the friends, the longterm friends, maybe the, even the casual lovers phase. And I am. Doing a confession to start off the podcast today. I am not an anti-vaxxer whatsoever. I am definitely team vaccine for the win. Definitely get your vaccine. But my confession today is that I just booked my. Vaccine and my first vaccine for next week, and I am terrified to get the vaccine. I don't want to do it. I want to do it, but I don't want to do it because have you seen how deep the needle goes into your arm? It looks horrifying. Anyways, this is Amy Tom, of course. And this is the hacker noon podcast. I am delighted to be joined again by Matt gross, the senior product marketing manager at Couchbase. And today we are also joined by Kurt Gratz. He is the co owner of C K H consulting. And today we are going to be talking about. No SQL versus SQL databases. So guys, thank you very much for joining the podcast pathologies for my vaccine rant to start off the show today, but I just felt like I had to get into it. Have you guys got the vaccine? Yeah, I can put your mind at ease. It is the smallest needle ever. And you barely feel it. It's not a big deal at all. I actually had both doses. I coach cross country and track at our local high school. So I was able to get it a little bit early because of being around some youth. So, I can. Put your mind at ease is that it's a very small, very easy shot. Okay. That's good. I'm glad that my mind has been put at ease, although I don't know if I believe you because I've seen the needle. I think Amy, that the difference here is that I have, let's say a lot of extra padding. And whereas you are in very good shape. So perhaps to you, the needle looks very large, but to me it was. Basically nothing. Okay, that's good. That's good to hear. Yeah. I think in the U S you're a little bit further ahead than we are here in Canada. I'm from Vancouver. And so I just got the email yesterday saying that I could book the vaccine. So I jumped on that. And I am. Both looking forward to it and also not looking forward to it at all. So that's where I'm at with that. Cool. So Kurt, I would love to have you go over your experience with Couchbase and databases. Yeah. So, myself I've been in the. Kind of data modeling and architecture of data world for quite some time. And it's always been something I've been super passionate about. And I kind of was able to make that transition of what we're talking about today. I used to be very heavy sequel and you know, that was where data lived. It was always in SQL based database. And eventually as time went on, we've kind of learned that. You know, maybe there's other ways or better ways to store data. And so I was, I've been part of that transition in my career and where, as soon as we started using no SQL databases, we were looking around at some of the different providers and then instill Couchbase is definitely a top tier in any type of no SQL database that you're looking at. I think they offer great features, you know, one of the things that really turned us onto it as a company first was their use of clustering. I think there they beat anybody in the history by far and how quickly and easy it is to scale horizontally. Which is one of the big reasons you're going to switch to no SQL is that horizontal scaling versus, you know, with most traditional database, just traditional SQL databases, it's you know, you can scale, but mainly vertically, right? You're just going to throw more hardware into one big box where with the no SQL options, it's a lot easier to scale horizontally. And Couchbase, in my opinion, is, was the easiest to scale horizontally. And being a user of the product, I would love to get your opinion on what kind of databases slash companies slash industries would be best suited for a no SQL data base versus using a SQL database? Well, that's a good question. I'm always a big proponent of the best tool for the best job. So I don't know that there's any specific like industry. Or anything that would like not be able to use a no SQL or Couchbase database for parts of what they're doing. But they do need to take into consideration what, you know, what am I, what kind of questions am I asking of my data? You know, maybe you're looking at some data warehouse type questions where you're really looking at big data and maybe SQL database with a cube. And you know, some of those more advanced technologies are better suited for that type of question and then closer to your users and your data, no SQL databases where you would store that. So I think it's not a either, or I think it's sometimes both and Right. And Matt, what kind of considerations would you need to think about if you're deciding between SQL versus no SQL? Yeah, so I think Kurt really kind of, already nailed the main driver for the no SQL is the scaling. And that's kind of the original purpose of no SQL in the first place was to create a database that could be easily distributed as a trick. Distributed database, a cluster database. And so I think in a situation where you have a large set of users, a large set of concurrent operations, a large set of data that's going to keep growing and those operations going to keep growing. I think a no SQL database is a great choice there. And I think one of the things that Couchbase is doing is especially even with the Couchbase seven with currently in beta is trying to make the transition from relational to no SQL a smooth one for the developer, you know, offering familiar. Constructs and structures and languages and tools and APIs that can make that transition a lot smoother. If you know, when, if, and when you need that level of scale and performance. Yeah. And I'll even I'll give a little shout out to one of the newer features in Couchbase on the, I had mentioned, you know, kind of depends on the questions are asked asking, but Couchbase in their latest couple of versions now also has an analytics role for their cluster that can answer some of those more complicated ad hoc, maybe big query, big data type questions. So just to. Shout out to them for that nice new feature to kind of keep your data. Maybe even more in one spot. Matt, is there a situation where you would not recommend Couchbase or no? SQL? You know, I think I would always recommend developers to try and learn the cool new things. Right. So always give Couchbase to try download it, check it out. But, you know, I think you know, if your use case one that, you know, you don't have a large amount of data, right? If you don't need that level of scaling, you know, a smaller database is gonna be just fine. If you don't expect your data structure to change very often or need to adapt very often. Then again, you know, a small set that's not going to change very often is going to be a fine use case for relational. And probably going to provide adequate performance for that. But, you know, once you get into sort of the more, you know, web facing public website, large use cases, you know, with personalization then I think you're going to find a lot of. Benefits or at least the very least a lot fewer headaches when working with a scalable, distributed, a no SQL database. So if I've started off on a SQL database, what are some of the challenges I might face when migrating over to a no SQL database? Like Couchbase. I saw I can even answer here. I think Kurt's going to have some practical advice too. We actually just did a presentation on one of the aspects of this. And I think is been kind of one of the hurdles is data modeling is going to be, is gonna be different from relational world to the no SQL world. Cause we're going to be modeling data as Jason instead of as. No rows tables and columns, but I will say along that front, you know, Couchbase seven is adding some more structural features in there to kind of, bring the familiar, you know, tables and schema level organization to a no SQL database. So that it's not exactly the same. Right. It's the Couchbase is not just reinventing relational, but it's providing those same types of structures. You know, instead of a table, you have a collection instead of a schema, you have a scope. And so on that can reduce the level of challenge. At least entry-level of moving over to no SQL. Yeah, Kurt. Do you have any experience with migrating from SQL to no SQL? We do. In fact, that's something we've done a lot over the years and you know, I think for. For developers, one of the biggest hurdles. And we talk about this a little bit in that presentation that Matt had mentioned is it's a bit of a paradigm shift on how you're thinking about how you do things. So now you can think more like a developer in terms of, okay, how does, how do I want my object to look you know, how should this. You know, react with my application, not how should I model my data and then how do I build my application around it? It allows you to no sequel kind of allows you to have that ability to think object first, think a user presentation first. And then how do I store my data maybe second and still get that same performance out of it, which is, I think one of the biggest. The managers, but that's a paradigm shift for us as developers because you know, we, in, in a traditional SQL model, you're, you know, I don't want to go to the database very often. I want to keep you know, my data in the most normalized form as possible. And so making some of that switches there is difficult. And then also, you know, there's a lot of different ways to ETL data from a. A SQL database to a no SQL database too. And you know, some, there's some different challenges that can come along with those. But that's another tool that Couchbase offers, you know, from lots of different ways to import your data. Sorry, what do you mean by ETL your data? Yeah. So, it's cha it's basically transformed. I can't actually remember the acronym but transforming your data from one place to another, as an ETL operation. And it's basically taking that, you know, it's in one schema, one form and converting it to that other schema and that other form it's a complicated process. It can be depending on how, you know, on what. What forms your data was in before and how you're, how you want your data model, that the end of it. So, it doesn't have to be though. I think it's. You know, if you're converting and you're just, you know, you're making some decisions along the way. Like, what do I want to keep normalized? What do I want to have embedded in my documents? And if you make those decisions along the way it's really not that complicated to make that migration from SQL and SQL. So I don't want to say that it's complicated and scare people off. It's just something you have to kind of make sure you, you give some forethought to. Yeah, I am. I'm scared. It sounds like a scary process because let's talk about the conversion process of having to convert that data. Do you have to do it, you know, like a line item by line item, data set by data set, or is there something that can automatically convert the type of data from your SQL to your no SQL. So, the approach that Kurt's talking about here is one that, I mean, to many people, it does sound a little scary, right? Where you have to kind of change the structure of your data on the fly. And I think that ultimately is the approach. A good approach to taking the, taking advantage of a no SQL databases, unique capabilities, and modeling the data to support that. But I, I think one of the things we're focused on this year with Couchbase seven, which is still in beta, and it's probably one of the reasons Kurt hasn't really done any projects with that yet, because it's not. You know, a hundred percent production ready yet. It's still in beta. But I think one of the things I'm working on is a project that makes the literal translation, automatic literal translation of a relational database to Couchbase taking advantage of those new structures, the scopes and collections. So it is not a situation where you have to go through line by line and figure out, do I need to transform this data or not? It's just doing a kind of a lift and shift. Of the data into Couchbase. Now the benefit of this is that it's super easy, right? It's just a script you can run and copy data over and bam, you're done. You're a no SQL course. The drawback there is that you're, it's still in relational format, so it's still in pieces. So you still have to join it. You still have to do asset transactions, all that kind of stuff is still there, but you can, it's a good platform for you to start building and you can then start to make changes and start to consolidate data when it makes sense to. To hit the performance goals and, you know, just have your model take advantage of that flexible Jason model and what gets converted versus doesn't get converted. Or maybe the better question is what needs to be converted and what doesn't need to be converted. All right. So I'll also have my little project, my little experiment, but it's converting as much as it can from, in my case SQL server, but the same approach should we take in for the other relational databases? So, the structure of the data where you have, you know, you have a sort of a database which contains schemas schemas, which contain tables, tables, which contain rows and all that stuff needs to go over into Couchbase. It'll be in, in different structures, but there'll be so similar. Right? So at Couchbase you have a bucket bucket contains scope, scope contains collections collection contains documents. So it's that same hierarchy there. We can just copy a row of data over into a Jaison document. And also, you know, things like indexes are important to copy over if we can users and authorization. Authentication well authorization mainly should we copy it over as well? There's things that, you know, might be very tough to automatically convert and we'll, I think we'll, we can talk about some of those but I think for the most part, that's a good way to start for very least a proof of concept, if not a good way to just, you know, get to the simplest model we can in no SQL and then start working. And Kurt was your experience. Did you, was there things that you had to manually convert over? I don't think there was anything that we did manually. I mean, we did have to maybe do some programmatic things to get things, to transfer over. But maybe, but not really manually. There's more. Okay. We're going to write us a script to transform this data as it you know, pull it out, transform it, push it in. I'm interested in Matt and some of those w you have to show me your toy you're playing with there. Yeah. I'll give a link to it here towards the end of the show. Cool. Thank you. And what kind of technical knowledge do you need to be able to complete this migration process? I think like the ones that I've done in the past is really just, you know, if you have some database knowledge, I, you know, I on both sides or some data modeling knowledge, that's really all you need. You know, really it's about if, you know, if you're doing a migration, chances are, you know, your data. And you also know your, how your users or applications interact with that data. Those are the two biggest things to know. And from there, it's, like I said earlier, it's kind of just about making decisions of how you want that data to be after your migration versus how you're storing it now. And those are, you know, maybe based on performance, maybe it's based on ease of use, you know, those are things that you're trading off as a D as a developer, when you're making those decisions. And Matt, what about schemes? Because as I understand, no, SQL is schema lists. So do these schemas migrate over as well? So, this is also kind of something that Kurt and I discussed recently. So this is great, but it's oftentimes it's called schema less because the database doesn't really enforce schema the same way that a relational data database does. Right. You don't predefine your fields in advance. You don't, you're not limited to just the fields that you've defined ahead of time. But I think a more accurate term would be schema flexible. Where you still want to have your data have, you know, a pretty well-known structure to it. Right. But one of the benefits here is you can, now you can have some flexibility there. So the documents don't always have to look exactly like each other. As they do rows a, in a table, right? So you gain some flexibility there. So, what that means though, is that the schema enforcement is kind of pushed out of the database into you know, other parts of the database where you can program in some enforcement or some checking, or even into the application where you put some of that put some that functionality, which, you know, for the most part is usually there. Anyway, there's usually. And we're not just relying on the database itself to enforce a business logic and structure of the data. So it's really just kind of a shift in you know, how much we depend on the database itself versus the applications that you use it to, to enforce that schema. And then it was, is that going to be the same kind of thing with something like stored procedures where you're really using stored procedures, more in the SQL format versus the no SQL database? Yes. So store procedures. I get asked about this time to time. The developers I talked to store procedures are controversial. I guess you could say, I don't, if that's fair to say or not Kurt he's smiling. Because you know, people have various experiences with sort procedures. I came from a company where anytime you wanted to touch the database as a developer, you had to go through store procedure. And I found that process just excruciating it's slowed down everything it was. I mean, there wasn't really any point to it. So based on that experience, a lot of developers are like, I hate store procedures. I'm never going to use them. But on the other side, you know, there is lots of good reasons to use for procedures. You know, if there's a complicated, a bit of data manipulation, you don't want to have to redo that over and over every time you touch the database. So it does make sense. To use them. But I think sparingly, I'd be curious to know a Kurt's thoughts on store procedures as well, especially when it comes to new SQL. Yeah, well, yeah, I've also gotten asked and I, what I've seen, what I've seen people do that maybe is frustrating to me a little bit is they end up bearing application logic into a stored procedure. So they're doing things that maybe really should be done at the application level. Some business rules, et cetera, and that storage procedure just because it's, you know, whatever it's close to the database, it's maybe what they needed to do. And the other thing that I see stored procedures used a lot for are things like triggers on a database. So some field changes in the database, they kick off a stored procedure that does whatever. Work needs to happen after that. And you know, no college Bates at least has a solution for that. They have a venting on documents if document changes or whatever, you can also kind of kick off that same. So there is a little bit of the you know, kind of one-to-one there depending on what you're doing with your stored procedures. So in that, you know, we were talking about migration, I would say you're either moving that sort of procedure logic to the application. Where it really should be, or you're using something like a venting to kind of keep that same logic in the same place. Right. And also something in Couchbase seven is something called a UDS user defined functions, which again, it's not a store procedure. Cause it's basically just a way to define a function. That lives in the database. And that can take the place of at least a subset of store procedures and the way they work. So there is some options in the SQL world and Couchbase world for putting functionality in the database. The other big thing that I see people do use stored procedures for is because they need to eat more performance out of their database. So a lot of times what a start procedure will do is write it. We'll write that database plan. Already, and you're just passing in parameters and it just has to change the parameters on the backend of the database. And that's not necessarily a concern that you have with a, you know, when you're migrating to a no SQL type world, because you're already, you already passed that performance. You're already kicking that way out the door. You're already 10 times or more faster doing the same operations as you were before. So, you may find it. You know that you really don't need that in most cases and speeding on the way the database operates. What about like acid and atomic operations? Is that a consideration as well? So this has been again, I've been with Couchbase for over five years. This isn't one of those questions I could ask all the time. Like what about assets? What about transactions? And yeah, historically that's been a limitation of no SQL databases is because of the distributed nature. Acid is very, very hard to achieve. It's a complicated problem to solve just as is. And you've put in the network into the equation. It becomes even harder. Unfortunately, there's been a lot of hard work in this area, in the industry and at Couchbase as well to bring asset transactions to Couchbase. So that capability has been there since Couchbase 65. And we're expanding that even further. And Couchbase seven. To make it part of our SQL implementation as well. So, acid is there now when you need it in a relational database, now, if you're doing the conversion program, like the little toy that I'm working with us is going to be very important, but if you're going through the thought exercise and the more thoughtful modeling that Kurt's talking about, the, you don't necessarily need asset as much because your data will be consolidated into a single piece as opposed to the multiple pieces in the relational world. Right. That's exactly what I was just gonna say is I think that we as developers, we're so used to relying on the database to solve those acid type problems for us, but we didn't necessarily, always need to have themselves. So it's you need to really ask yourself, does this use case need that type of like a transaction? Maybe you're editing a user's name and it really doesn't need, you don't really need a transaction for that. You don't really need the roll back. Now, obviously, if you're doing a you know, maybe a checkout situation and you need to you know, make sure that a balance for a gift card is always that correct balance and then rolls back at that doesn't happen. That's a very good use case for an acid type transaction. So I think it really. You know, just need to limit your use cases because that also will help your performance when you're only doing that when you need to. Right. Makes sense. Okay. I want to talk about what happens when shit hits the fan with the migration process and don't lie to me and say that it doesn't hit the fan. So what are some things that can go horribly wrong with the migration process? So I'm gonna, I'm sure Kurt has some great stories, but I want to talk about some migrations I've done in the past, and isn't necessarily a justice SQL to know sequel thing. It's any sort of migration these kinds of problems can happen is okay. I'm writing a ETL script. As we mentioned earlier, that stands for extract transform load where I'm moving data, maybe changing it and then putting into another database. And w as I'm developing that script, I may be using a small sample of the test data because I don't want to have to run it through every single row of the data just to do my development. Right. So I'm using a small sample and so everything's great. Everything's great. I'm ready to run it into the production server and run it on all the data. And I think one time I'd done this and I ran into an out of memory error because I was basically loading everything into memory. And for my test script, this was fine because it was a small sample. So what I learned from that day is basically, you know, add lots of logging to the process make sure that you expect your scripts to fail. And you can, you know, that, that point on you, you can just build it as if you're going to fail, something's gonna fail and when it can restart it and after we make a fix, we can keep trying it over and over. It's not ever going to be a one and done type of situation. Yeah, well, first day me, I want to say this is a great question because w people often are like, you're stuck in that happy path. And that's the biggest thing in my experience when you're doing any mite migration is you really have to make sure you're not just thinking happy path, because there. If, when you're S when you start, like Matt had a great example, when you start with that samples that you're like, Oh, this is all the happy data, but when you get into the real production data out there, you know, I've seen everything I've seen people's names are emojis. Did you handle that? You know, The state is pizza. Did you handle that? You know, maybe the system is 15 years old and there was no validation at that point. There is now. So you're like, Oh, I don't have to worry about that. Well, you're not thinking 15 years ago, you probably weren't with the company 15 years ago. Those are the types of things that happen in these type of big bulk migration scenarios. So like Matt said in your script, you're thinking about, okay, basically you want to catch every scenario and. Keep the script moving, have the ones that failed and you need to log that out and you need to have then a plan to go through those ones that failed. How am I going to get the state in? Do I care? Maybe it's, you know, maybe this is a good time to remove everybody that had pizza as their state. You know, so I think that is important as you're looking at it. Any data can basically be in any format that it wants. Like I wouldn't you know, count on it always being in Zulu time. You know, when you're doing times on. So, you know, like again, you're probably migrating something that's been there for awhile. And maybe the rules are this way now, and maybe they're standardizations now, but unless you really know the history, they might, it might not have always been that way. That's been my biggest thing that I've noticed over the years is everyone you run into. Something that you didn't expect in terms of how the data was stored or what data is actually in the database? Yeah, I think this is also a great point. We were talking about, you know, no SQL being schema-less or scheme of flexible and, you know, sometimes that's concerning for a developer. Cause you're worried about the data becoming, you know, maybe unruly or invalid, but in a relational database, I'm sure you've seen this before Kurt using a VAR char field. To store a decimal or something like that. And it could easily be corrupted with, you know, a non-numeric character and that, that same sort of thing can happen in relational as it can in no SQL. So it's really just a question of you know, handling all those weird edge cases. No matter if it's a relational or not. Yeah, I've seen, I actually have a story of that with so Java float error, right? It's was supposed to be all integer based and instead you've got, you know, one with 14 zeros and four the end and you know, again, what do you expect versus what is actually, there is often not aligned. Yep. Yeah. I think even like the word migration just makes my skin crawl. Like it just is such a scary process. And so when you're going from no SQL to SQL, is there a chance of losing your entire database in the process? I mean, I certainly hope not. That's, I mean, probably you shouldn't run it for the first time on production, I think running on a copy of it to make sure it's not going to destroy anything or correct anything. Absolutely. I will also say one of the points I wanted to make. We've been talking about migration a lot. And, you know, there may be situations where you want to leave behind and your SQL server and go to Couchbase and just, just ditch the old database and bring the new one on. But there's lots of use cases where a migration doesn't mean we abandoned the old database. We may want to. Copy or synchronized data into the no SQL database for certain use cases. And that's a place where Couchbase has done a lot of work already. And so it's, it's not necessarily about leaving the old database behind. You can certainly, I think even Kurt said this early on, you can use multiple databases to accomplish, you know, the use the best tool for your use case. So, I know migration may sound scary, but I also want you to think about it as not necessarily. Abandoning, but building a bridge between the two of them and not burning that bridge. Once you cross it. I don't often trust my migrations, but when I do that, it's in production, man. And what about the process of migrating from one, no, SQL to another, no sequel. I know you can have multiple different databases, but is, I mean, I'm sure that there is a situation where you would need to migrate your no SQL to another Virgin of a different, no SQL. So is that process similar? I would say yes, it's still similar. But it generally is going to be an easier because you're, you know, you're gonna. Basically be in either likely, still a document store. And you're kind of just moving the data that lift and shift that Matt's talking about in Couchbase seven that they can now do from a SQL database to a, to Couchbase it's really that lift and shift, just move the data over. But again, if you're asking yourself, why am I migrating? It's hopefully, you know, has some reasons besides just moving the data. That you're doing that you're going to want to run it through and maybe either adjust schema or do things better or more intelligently. Yeah. And I would also say, you know, we're we've we used the word, no SQL a lot this episode and no SQL is really a broad set of different types of databases. Right. I think we're talking about Couchbase and Jason were talking about document databases. So going from documents, that document is going to be a little more straightforward, but if we're going from documents to graph or from relational to key value, I mean, those are going to be. Pretty significant transformations that, you know, again, I would definitely go back to the question, why are we doing this? I'm making this kind of change. But I think the document databases is a really good general purpose analog to relational. A lot of the same concepts there are familiar. So, you know, that's something to keep in mind when you're moving from database to database. Right. So I'm hearing a lot of good things about the migration process. Like making sure you test before production, but what are some other, what is another piece of advice that you might give to a first time Migrator? To me it's about thinking, you know, making sure you're taking that step back and thinking through your. Use cases. We've mentioned a couple of times why are you doing what you're doing? You know, getting, you know, whether that's you and your, the business, or, you know, you have business users involved and that you're doing this for a. A specific and valid reason. I think sometimes like, you know, developers, we have shiny object syndrome. It's like, Oh, I want to try this new thing. But if you're going to do this type of op you know, full migration it needs to now be on purpose. That would be the advice that I guess I would maybe say. Matt. Do you have a different piece of advice that I think that makes sense. I think what I said before is the key for building any sort of migration tools or script is just expect failure. Failure is going to happen. If actually, if it actually succeeds, you know, be suspicious of that, like, well, why did they go right? The first time I tried it, you know, make sure to have plenty of logging. So, you know, when it failed, why it failed. And make sure, you know, you expect to run this thing multiple times. So, if you get into a situation where it's. You know, you happen to go through a lot of manual steps to restart the process. Well, maybe you should script those out because you're going to be running them over and over and over again. So, you know, try to take those manual steps. This is what a lot of developers do, right? This is why bill or developers is because we're taking stuff that you have to do manually and scripting out, running a program to do it. So let's just general advice. I would say. And building ETL is migration processes. Awesome. So Matt, this is now your third time on the podcast. Maybe a hacker noon or record. And even though you've been on the podcast so many time, is it please tell the listeners where we can find you online? Sure. You can find me on Twitter. Groves, that's an S at the end, you can email me Couchbase Matthew with two Ts dot groves@couchbase.com. I also want to give a shout out to this tool that I mentioned earlier. It's on GitHub right now, get hub slash M groves, and it's called SQL server to Couchbase. So if you're interested in kind of trying to try that out, playing with it, making suggestions, I am I would love for you to even just come by and create an issue and says, Hey, can it do this? Is that might be enough reason for me to say, Oh, it can't, but maybe it should. I'll go ahead and start working on that. So I would appreciate any sort of feedback on that project. I will put that link in the show notes and Kurt, where can we find you? And C K H online. Yeah. So, my Twitter is also my last name, Gratz C so it's grots and then my first initial C that's also on my LinkedIn. And I think my Facebook also my legal legends summoner name, if anybody's a legal legends fan out there or my email is Graziano at CKH consulting that calm. And that's our corporate website, CGH consulting.com. And I also blog out there. I've got a couple of good Couchbase blogs out on there. If anybody's looking for. For help. They can certainly reach out if they, you know, if you're doing any of these ETL things and they do sound scary. Certainly give us a shout out. Well, I will give you and our listeners in general and in rotation to write on hacker noon as always to reach some new contributors or to reach some new readers. So thank you, Matt and Curt very much for joining me on the podcast today. I very much appreciate it. Thank you very much, Amy. Yeah. Thanks for having us. It was great. No worries. If you liked this episode, don't forget to like share and subscribe and tell all of your friends. This podcast was produced by hacker noon. It was hosted by me, Amy, Tom, and it was edited by Damian. We will see you again later on goodbye.

Migration Makes My Skin Crawl: From SQL to NoSQL

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

A Snap Is Worth A Thousand Words (Podcast Transcript)

6 Database Migration Tools For Complete Data Integrity & More

A Snap Is Worth A Thousand Words (Podcast Transcript)

Dealing with Data Migrations on Multi Tenant — Laravel

Cloud Migration Strategy: An Overview of the Process from Preparation to Implementation

How To Copy Terabytes of Data Between AWS S3 Buckets

A Snap Is Worth A Thousand Words (Podcast Transcript)

6 Database Migration Tools For Complete Data Integrity & More

A Snap Is Worth A Thousand Words (Podcast Transcript)

Dealing with Data Migrations on Multi Tenant — Laravel

Cloud Migration Strategy: An Overview of the Process from Preparation to Implementation

How To Copy Terabytes of Data Between AWS S3 Buckets

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps