What The Shell?! (Podcast Transcript)

Listen to the Hacker Noon Podcast on Apple Podcasts, Spotify, Google Podcasts, Stitcher, or wherever you listen to podcasts.

So it turns out there's more to Shell scripting than Powershell and Bash! 😱 Amy Tom chats with Michael Nitschinger, a Software Architect at Couchbase, and Jonathan D. Turner (AKA JT), the Co-Creator of NuShell, about Shell scripting, how to explore data interactively, and all the different ways to query your database. Amy finds out the use cases of Shell scripting databases including editing sensitive data records and more. ✨

In this episode, Amy, Michael, and JT get into:

• What's the deal with NuShell? JT explains he co-created NuShell to be a FUN way to use Shell 🥳 (02:38)
• How easy is NuShell - a scripting language - to learn and why would you learn it as opposed to sticking with Bash or Powershell scripting? JT says it's got Typescript flavor with command help and less bullshit (not a direct quote 😂 ) (07:39)
• Using Shell scripting to modify your database workflows, such as looking at the health of your indexes or importing/exporting data ↕️ (15:58)
• Michael says, "with more flexibility comes potentially more complexity" They chat about the different challenges between building a full-blown rest APIs vs a simple data export 💪 (19:51)
• How do you determine what kind of query you're going to run on your database? Michael talks about all the different ways to query a database, and Amy asks a million questions about querying (23:50)
• Why is Couchbase integrated with NuShell and not Powershell or Bash? 🤨 (33:44)
• What else are people using NuShell for? (37:48)

Follow Michael & JT:

1. You can follow Michael on Twitter @daschl
2. You can follow JT on Twitter @jntrnr

Shownotes:

• You can check out NuShell on GitHub: https://github.com/nushell/nushell/
• You can learn more about Couchbase Shell at https://couchbase.sh/

Machine-Generated Podcast Transcript (Please Excuse The Errors)

Amy: [00:00:00] Ooh, is that a whiff of summer that I smell it in the air. It is heating up. Things are getting crazy. Vaccines are flying and we are prepping up for summer. So I'm excited and it is almost June. But of course, You are listening to the hacker noon podcast.

And my name is Amy Tom. Today. I am joined by JT, who is the co-creator at NuShell, which we will get into a little bit here. And also by Michael, who is the software architect on the SDK team at Couchbase. So welcome to the podcast . JT, how are you doing today? 

JT: [00:00:38] Doing good, right early here in New Zealand, but yeah.

Amy: [00:00:43] Where in New Zealand are you from? 

JT: [00:00:44] I'm originally from the States, but I've been living in New Zealand the last four-ish years. 

Amy: [00:00:50] It's beautiful with the accent, the American accent. Michael, how are you doing today, Michael? 

Michael: [00:00:58] I'm doing great. Thanks. So very much for asking, where are you based Michael?

I'm based in Vienna, Austria. So pretty much middle of Europe. Yeah. 

Amy: [00:01:07] Amazing. And I am in Vancouver. So we are in a worldwide zoom call today. That's crazy. All right. Awesome. So today we are going to get into some scripting stuff. So I'm excited to hear from you  about scripting and JT. Of course, being the co-creator of NuShell is going to be a scripting expert. So I guess we'll just start right up from the top. Why would you want to use scripting? What's the purpose? 

JT: [00:01:34] I think one of the first things, when you pick up something like Python or you pick up NuShell the first thing you want to do, or just simple ideas, like I want to add some numbers together.

I want to run a command and get some output. But the more that you become expert in what commands are available, You, I think your ambition grows from there. You want to take that one command, then grow that idea into maybe a pipeline. All right. I want to take the output of this command and put it here and then change it into something new.

Then take that and change it again into something else. And that way of thinking of transforming data from one thing to something else. I think naturally leads itself to growing into scripts. We were like, Oh, I'd really to reuse this idea over and over in different contexts, maybe for different directories or, I want to reuse it in some later point in time.

So I think scripts are a way that we can capture that idea, that experimentation, bring it back up and then use it again.

Amy: [00:02:35] The questions that I have for you are around NuShell and why you would want to create another shell. You know what I mean? Because I think I guess I'm not really sure about what the competition is like in the scripting space, because I'm only really familiar with bash and PowerShell. 

JT: [00:02:54] Yeah. And I think that's most people, they come to it and they say well, like I want my shell when I need to.

And one of the things. That we were thinking about when we first started working on NuShell, is can we make it more comfortable? Can we make it more fun in terms of like you go into a shell? But you want to be in the shell, like it is an experience of actually an inviting experience. 

Amy: [00:03:19] Yeah. I don't know if I've ever correlated fine and shell, 

JT: [00:03:24] but that's yeah.

You're not alone. That's going to be most people. They're not going to jump in and say, Oh, this is great. This is like a, my warm bath of the terminal, but why not? Why not? Why not make it feel so ergonomics so nice. We're looking at even the error messages should feel inviting.

That makes you want to go and fix whatever it is because they look attractive. They draw you in. 

Amy: [00:03:48] you're telling me new shallows, the warm bath of shell. 

JT: [00:03:52] Sure. 

Amy: [00:03:54] I sounds 

JT: [00:03:54] inviting. That's the ambition, that's the ambition. One of the when we first started. We had this kind of theme that we wanted to go for, which is, fun should be one of our goals.

When you look at the top goals of the project, fun should be in that list. And that's, not the first thing that you think about when you think terminals are shells. So that's really where it began. And then the other place is, NuShell thinks about data in a structured way.

It wants you to engage with the commands in a way that. That really elevates the Unix way of thinking. So the Unix way of thinking is I have a command. I pipe the output to another command. I have the output of that pipe, that to something else. And it's all texts that flows between them. But what it does let you do is compose your commands together in NuShell, all the data is structured, so you never lose the information.

If I open a Jason file or I open a CSV or Excel spreadsheet, that's all structured data in there, their columns or their objects or their fields. You don't want to lose that. You don't want to go from that back to texts and parse the text again, come back from that to something else you want to just say, open this Jace on file, go to this place, get the username or something like this, get the username out and just have that as one simple.

Yeah, openness pipe to something that can get the name, pipe it to something that can print it out in a nice way. And all that composition, as you begin to get used to new shelf feels so good that you can just open something up. And then the next command that you run. Use all the same style of commands.

So whether you're opening and chase on file or you're, getting some data from the web, you're downloading a URL and you want to get the contents from some website or you're, you're looking at the, just the files on your system, all of those work, the exact same way. So it's really good at building muscle memory.

Michael: [00:05:53] If I can inject here quickly, please. Basically what you were talking about, the fun part. When you've been working with computers for a long time, it's not often that you come across some piece of tech where use it in the first time, the first couple of times, and you'll think this is really different to what was there before.

And we can talk about the rust in the same context to me, but NuShell is something similar where. First you think it's just a shell, but then you start using it. And I know fun is a. It's not, it's hard. Fun is different for everyone, right? But once you start using it and you discover the power of those different commands and how you can compose certain things together, it just makes you feel so much more productive and so much more powerful than your shell, which then makes it much more fun to interact with it.

So as. As was mentioned it. So you open a chase on file. You extract data, you store it as XML, right? So you can do that in 30 seconds. If you know the commands and previously you will probably. It would take you a much longer time to do this, right? So the whole obstruction and bringing structure to the different components, in my opinion is really a game changer when it comes to data transformation, ETL pipelines, which then we can preach towards Couchbase database structure, et cetera.

But Neushul alone with the functionality it provides as a day-to-day shell. Once you get used to it, it's really makes it so much more productive. And then as you grow to learn, it also makes it much more fun to interact with it daily. 

Amy: [00:07:17] Yeah, definitely. And so one thing you mentioned, Michael, is that you need to know the commands first and then it will be faster.

So with something like NuShell, which is, I don't know the right terminology around it, but uses the programming language rust. Is it easy for someone to pick that up? 

JT: [00:07:37] So the, I guess you could think of it in two ways. So there's the language that NuShell is written in, which is Ross. So all the implementation behind the scenes is all written in rust.

Partly that's an artifact of my background. So I helped work on the Rus program and language when I worked at Mozilla. And so I'm really familiar with it. But it's also, when you look at the languages that you would pick for such a project rust is what you would pick. It's extremely fast. It's extremely secure.

So if I have something running, sitting in really like vulnerable places of people's machines, you want that to either want to, you want to remove any possibility of any security anything in those situations. So Russ is a natural. Implementation language. And then NuShell itself is a scripting language kind of designed by, the careful hands of the folks putting it together.

And that is really a kind of an ergonomic language that builds from the structured pipelines. And we were talking about earlier. So it takes that idea and says, Let's make a scripting language that feels like so I also worked on TypeScript as well, so it has some TypeScript flavor to it. So you can come to it from other languages and there's enough there that it doesn't feel alien.

Like you can get in and learn it pretty easily. As far as the commands, the system lets you ask about the commands. So you can say help space commands and begin to learn NuShell interactively inside of NuShell by experimenting and saying, okay, what about this command? And then it'll pop up and it'll have some examples and some texts so that you can learn the command at a time.

And I would say like the last thing about what NuShell is like to learn is that. And Michelle is in immutable by default. So you're not changing anything. You're not, you're not messing up your system, if you could think of it that way, you're just running commands and seeing what the output is.

And if you like it, that's great. If you want to turn it into something else, that's also great. So you're interactively exploring. Your data or outputs of things without having to really worry so much. I think that also feels like a bit of a game changer if I could use the term because you're not having to learn a whole bunch of language upfront.

You can learn a single command, explore a single command and then learn one more. And as you learn those two, maybe you can compose them together and get more power that way. 

Michael: [00:10:07] Also sorry. So also the command itself, many of them usually you already familiar with. So for example, if you are familiar with the LS command on a unique style system, NuShell also has a LS command, right?

So many of those commands that you already know from bash, other shells are already there, but with a different flavor structure, et cetera. So it's not like you're not going from basically from Pesh to New Zealand inspectors. They want basically of learning the shell. Many of those commands will feel very familiar and natural to you already.

If PowerShell, if Peshat cetera. 

Amy: [00:10:39] And I want to talk about choosing what scripting platform to use, because like I said, PowerShell and basher, obviously like the two biggest ones. And I wasn't even really necessarily aware that there were other options. So I guess my question is twofold.

what's really the main difference between NuShell and bash. And how would I go about choosing what scripting platform to use? 

JT: [00:11:05] So really NuShells ambitious in the sense that it's not. So when bash has created, I think bash was created first to be an interactive experience and then thought about okay, I can make bash scripts.

It was kind of a second thought it wasn't designed. That would be the original way that people would use. It would be both styles of experience. I think PowerShell is the same way, so it was more interactive, but the language grew up from there. It has some quirks as a result.

And what we wanted to do with NuShell is to say from day one that we know we want to be able to do scripts, like actually write full scripting files, but also. You can just jump in the shell, right? The exact same style in the shell itself. So I think that's one of the reasons why it is its own language is because we wanted a language that is actually naturally scaling up and scaling down to whatever size you need.

Right. 

Amy: [00:12:04] are there a lot of different options to choose from when it comes to deciding what scripting platform to use? 

JT: [00:12:11] It is the scripting language. You can run for example, Python or Ruby or anything on top of NuShell, if you wanted to. But NuShell itself is already a scripting language.

So you don't actually need to call into other scripting languages. You could, if you want, just stay inside of NuShell itself, use it, write your scripts in it, grow. And in fact, grow programs ultimately in NuShell itself. 

Amy: [00:12:36] And what variables is NuShell using 

JT: [00:12:38] Oh, so I guess one of the, one of the interesting things in NuShell is because it's both a shell and a scripting language.

We have to sit one foot in each side of the world. So in the shell world, you have to work with environment variables, and this is how commands. Communicate with each other basically, but you can set some environment variable, and then as you run another command, you could detect that environment variable in your environment.

So we do support that, but NuShell also has the native variables that you could just set, a little variable to some answer from the output of a command and then use that leader. 

Amy: [00:13:17] Okay. All right. No. So let's talk about then using shell scripting to modify the database. Michael, can you tell me what the correlation between NuShell and Couchbase?

Michael: [00:13:32] Yeah, sure. So when I started playing around with Neushul early on, so one of the early releases, I quickly realized that. Having the power of, and flexibility of NuShell, not only, scripting is one part, but also the interactive way to look at data, basically look at structured data or semi-structured data, like chase on these, taking that and combining it with a database like Couchbase gives you a rich tool set with Minimal effort basically on our side, because the thing is, as a developer there are so many things you potentially want to do with your database. Think of workflows, like inputting data, exporting data, analyzing data, or let's say you have two databases and you want to figure out what are the differences?

I have a production system. I have a staging where a development system, and I want to figure out what's the difference in the data et cetera, like those kind of workflows are. Potentially very simple, but also potentially very complex. So writing. Tools as a company like Couchbase writing tools that would support all these use cases in incredibly difficult, because there are so many, so the design process was okay, let's take the flexibility of something like NuShell and add the couch specific features on top, like commands, et cetera, which allow you to import export data on the one hand, but also for operations, look at the state of the database. What am I indexes on the database?

Like how am I, or like, how is he doing, what is the health of my system, which then gives operations or dev ops the same functionality and flex flexibility that the development developer would get. So taking a use case, let's say, I want to, if I'm a developer and I want to import data, I have a bunch of GS and files flying around in my directory, and I want to input into Couchbase.

So I with a couple of NuShell commands, I can load those files into notion and then use a coach be specific commands that we edit on top to take that stream of data. It's just a stream of data. We don't care if it came from XML. If it came from Jason, if it came from. Even a website or something.

So basically that data gets imported into NuShell. We take destruction and then store it into Couchbase and you can do it with a couple of commands. And because Neushul provides us the flexibility of. Transforming the data along the way you can, for example, perform data, cleanup tasks, or data enrichment tasks in your Nooshin pipeline.

So you clean up the data, you enrich it and then you store it into Couchbase and you can go the other way around. You can export data from Couchbase and then clean it up and store it as XML or story is chased home. Or if you are in operations maybe, and this is where scripting comes in again.

So then I can go in and say, okay, I want to run a script, which every three hours, for example, looks at the health of my indexes that are still in my database. And if one of the properties, let's say one is. Or have a certain quota or an index has been failed over and it's not available anymore, whatever it is.

So I can run that. And then I can write a new shift script that analyzes this data. And then I don't know, call to paycheck, or it gives me an email warning, whatever. So I can combine the flexibility of data input export and working with system state together with the flexibility of NuShell writing scripts, analyzing the data and transforming it in the way that.

I need for my use case. So that's the flexibility of combining both. 

Amy: [00:16:57] Okay. Michael, what I'm hearing you saying is that you use NuShell in the database to import export and also to modify the records. Occasionally for certain reasons cleaning up the data, for example, 

Michael: [00:17:13] yes. Yeah.

So it's not used inside the database, but rather the user gets a CLI tool basically, which happens to be a full blown shell to perform those tasks. 

Amy: [00:17:21] And I was going to ask then, why would you need a full blown shell for a data base in the first place? Could you not do those queries without the shell?

Michael: [00:17:29] You could. Yeah so of course, as a database, we provide SDKs for all the languages, Java, Python, whatever, but those are more used for you ride your application server in Java, and then you deploy it to production, that's where the regular is the case coming. But if I think of it from an.

Interactive way. So I'm a developer on my local machine and of course I'm writing my application that I'm going to deploy, but there are so many other tasks which I need to do separately of my main application. As we talked about data import data export, which very often, and it's the same with.

In my experience with many of these extract, transform load, those ETL pipelines, that most of the time you actually spending, cleaning up data. So bringing the data into the shape that you'll actually need in your database when you store it, or when you extract it, especially coach business, coach base doesn't enforce any schema.

The schema is enforced by the application. So we have an implicit schema. But depending on the application that you have, it could be that the chase and documents that you ingest are very complex and very different, because I don't know they are being ingested from 1 million different sensors.

And then you need to figure out what's in them. And then when you export the data, very often, you need to clean it up or you only need certain parts. And this is, I think this is, this brings us to another thing where. You mentioned performing those queries without the shell. Yes. But what it can do is you can combine it.

So we have a queer language, which is called Nikki that is very similar to SQL. So what you can actually do is you can run a nickel query, so you can write a select statement with choice, whatever you want, and then take the output of that resort, pipe it into NuShell and then perform more operations on it.

You can make it as simple or as complex as you want, or even combine it. Huh. 

Amy: [00:19:17] Can you give me an example, a real real-life example of using NuShell to modify a  file in the database or to clean up the data. What do you mean by clean up the data?

Michael: [00:19:28] Cleanup again, depends very much on the application or the data you have, right? So if you think of it from a relational side, so traditional relational database has a very structured schema. You have your table with, and every table has a specific schema with columns, et cetera. Now Couchbase, doesn't impose that schema on you.

So it's. It gives you more flexibility, but of course with the flexibility becomes potentially more complexity, right? Depending on how you structure your application, where your data comes from, et cetera. So you mentioned a use case where. And we I've seen this a lot in the past at my previous company where I was doing consulting.

Especially if you go into enterprises, let's say every department has their own database and the trouble starts when these different departments need to interact with each other. So they need to come up with API APIs or ways to communicate and share data between those departments. And of course, like building full blown, rest APIs, whatever that's the.

The highest sophistication, but very often, all that's needed is a certain data export, for example. So every day we're going to export data and send it over. To this other storage where the other system is picking it up. So of course you can go in and say, okay, I'm writing a child application for this, whatever, but you could use NuShell or CB shell on top, which basically you run a couple of commands.

You extract the data you want. And then maybe, I don't know, let's say you have sensitive information in your data that you can share with other departments. So then you have the data and then you can scrape the data and remove the sensitive information. Before you store it in XML or chasing whatever is on file and then send it over to another department.

So that's just use case that I just made up. But it's very common, these kinds of situations, or to give you another example. As a developer, one thing that's that comes up very often is if you just want to try out something is I need simple data, right? I need fake data, whatever.

Just something that I can test it out and play around with it. So in CB shell, we added a fake command, which allows you to generate fake data based on a template. So you can say generate my user with kind of random, first name, last name, email, address, whatever, but give me 10,000 rows. So then you get a stream of 10,000 users.

Which you can ingest into the database. So basically with one command, suddenly I have my database filled with 10,000 random users that I can then go on and create an index period, play around with the database. That's another use case that comes to mind. And there's a third one, so that's the development angle, but then from an operations perspective, You can also use our commands to monitor the health, inspect the health of the system.

How are my notes doing right. Coach visits, distributed system. It can have many notes. Some of them could be danced or you could say, okay, I'm running a chop, every, I don't know, two hours that inspect that this quota of my system to make sure that all the notes in the database and not over a certain quota to make sure my system is healthy is not overfilled, et cetera.

So I can. I can query my notes every two hours and then run some script code, which checks against certain parameters that I made up, let's say, okay, I want to make sure that it's not over 80% of my disk. And then when it's over that I can filter it and then I can get a list of all the notes that are.

Unhealthy, whatever that means in my context. And then I can take that and send an email to my service admin and say, Hey, please look at those notes because they look on healthy, something like that. So it's very flexible, but there are many use cases which are really driven by the very specific needs of that person, which is very hard for us to predict up front.

Right. 

Amy: [00:23:04] Michael, one of the things that I've been doing over the past couple of episodes is just learning about databases in general, because I've, I'm not a database expert. I don't come from a database background. So I've been learning through Couchbase employees, essentially of database.

Information. So one of the things that I've talked about with other members of the Couchbase team, and what I want to talk about with you is the querying of the data, because from what I gather there's so many different ways to query data. Whether it's going to be like a nickel query, whether you're going to run your NuShell script or whatever.

How do you determine what kind of query you're going to run on your database? 

Michael: [00:23:48] So that's an interesting question I even did. So we have an annual coach bus conference, coach bus connect, and I always did a talk about which service to pick, because it's it's a very well, yeah, but actually in my opinion, if you get the basics, which I can tell you about it's actually, I don't find it that complicated.

It's just that. If you see all this different services, you quickly get overwhelmed. But there are a couple of rules you can apply, which very easily allow you to figure out which service to use in Couchbase. And the reason why I gave the talk is because I'm working on the SDKs, which is really where the developer interacts with the system.

And this is where the challenges start. Let's see. So I spend a lot of time figuring out what right APIs are, the users are interacting with. Yeah that's what I spent most of my day. Yeah. Yeah, so I like the analogy of a tool belt, right? So you have a tool belt and the, your tool, but you have different tools, right?

You have a hammer, you have a saw, you have different things. And for different use cases, you pick different tools yet, right? You wouldn't really in a nail with a sore or something, that, that just doesn't work. So coach is a handful of services that you interact with as a user.

And it depends on from what angle you're coming from. Usually what I'm personally, what I'm telling users is, especially if they're coming from relational databases, they're very quickly drawn to Nikki, right? The queer language, which looks very much like sequel. And it is powerful. But again with power and flexibility comes complexity, right?

So that's why I always start with our key value service or data service. Which many people. Understand from a caching point of view, right? So you have a unique key in your database and you have a value associated with it. So if you know the document, it, the unique key in your bucket then going and which document to load and store going through.

The key value service is the easiest and fastest way into Couchbase. Storing a document, retrieving a document or retrieving multiple documents is really the fastest way. And actually, if you look into the architecture of, for example,  and the crew service underneath also calls into key value, right?

So it's just complexity on top basically. So again, if you have to document that the, you go through the Kybella service, that's the easiest and fastest way in and out of countries. Okay. That also, that's also probably recommend what I recommend is when you're building an application, you want to develop your database schema or the way your documents look like based on the hot coat path of your application.

So I know that let's say I'm building a specific application and I know that 80% of my users will go through that path. They go to that URL, they access those documents and I want to make sure that they get. The fastest experience possible. So I'm making sure that's the fastest way. So I'm modeling my documented, these et cetera for that pattern, but then okay.

Let's cover it. So I have a fast application, but then very often you have many secondary use cases where. It's not always say I need to load one document. And this is when Nicole comes in, where I can see, I keep in secondary indexes on my documents and I can query, I can basically Fitch information from the database where I don't know the document it, this is where I'm building indexes. And this is where I'm starting to write those sneaky queries with joins and predicates and where clauses and all those things. Nikki is always going to be a little slower than key value just because it needs to do more work. A query engine is involved.

It needs to do that. Planning the parsing, like all these things that seem pretty key value look up and I'm doing my colleagues on the service side that this service per saying, it's just a hash lookup when you do Cascadia, because there is so much going on under the hood. But if you do, key-value, it's really pretty much.

The ID, it does a big hash lookup. You get the document back, which is very quick, right. And a query engine just needs to do more work, but it's also more flexible. So that's the second year. And then we talk about search, right? So if you have full text search use cases where you need.

Freeform searches. You need faceted searches, right? You're building an online shop where I'm on this side. Like facets. Okay. I dunno, you're searching for, I dunno, clothes. And then you see the brands on the side. Like those kinds of things. That's where search comes in. In my opinion, if you need.

Search you need it right. There are certain use cases, which you can't do with Nicole because Nicholas Black and white, right? Cul you either get a restart or you don't get the result. If the career matches with search, it's more gray, right? So search says, Oh, this three, this string.

Matches 80%, but this other thing, only 70%. So it allows you to rank perform searches on big bodies of documents, where it's not black and white. So that's the search use case. And then. The other ones are eventing and on the lithics. Now, eventing again is another site topic where you can think of it, a little bit of stored procedures where you can run certain functions when documents change, when time is fire, right?

That's a little separate. Usually you use those for data enrichment tasks or other things that need to run in the background, right? You're not controlling this from your application, but. Again, talking about data cleanup, right? You could have an inventing function that runs in the background every time it sees a certain document type and it con it contains fields that, that should be there.

It can go in and clean those up, basically remove those fields, store that document again. And in the background, basically update your application. And then analytics, which we edit in six point, Oh, I think this is really where. So nickel is really used for online or LTP type use cases where you would have your traditional relation database.

So you, if you have your indexes tunes you get responses in milliseconds. But that also means you're working on your most recent and online data. That's what you want. But then you have use cases where, what people call it data Lake. So you have your. Terabytes of data.

And then once a month, someone from the I dunno from the reporting team says, Hey, I need to run this report over all my data. Which might take four minutes, five minutes, whatever. So that's, it's very separate use case. Usually you wouldn't use the analytics service as part of your intake application, but it's really meant for those use cases where I really want to run reports or extensive queries against my data.

The cool thing is with the analytics service. If before we had it, what people had to do is they use Couchbase as a, like the primary system to perform the application workloads. And then they had to export all the data into Hadoop or some other system, and then perform the analytics course there. Now with an analytic service integrated in Couchbase, that's all gone.

You can basically perform your analytical queries that might run minutes or longer. On the whole datasets on the same platform where you would do your key value nickel operations that are online services. That I think that's the power of the system. 

Amy: [00:30:51] Okay. But I feel like I still don't really understand, like what, w as a new database administrator, how do I choose which query option to use?

Michael: [00:31:01] The community itself. It's probably not defined by the operator, but rather by the developer. So when you develop your application, you extract the data and then you create the index or you just use the primary index. And this is where operation comes in and says, okay, actually it looks like this query is running slow.

And then let's see how we can create a secondary index that makes this queer perform better. But usually the way you interact with the data is. Defined by the developer who writes the application because they know what data they need and how they need it. Excesses, operation really looks after is my system healthy and are those queries going as fast as possible, but they are not concerned really with how the career looks like on the, how it executes.

Amy: [00:31:48] Okay. So as a developer, I would already be, I would already know going in that if my database administrator wanting to run this specific function, they would be using a NuShell script. 

Michael: [00:32:01] Not necessarily. So the operator has many different ways to do it. NuShell scripts would be one way.

But it's just something that we provide to the user as a benefit, right? 

Amy: [00:32:12] Yes. What I'm saying as the database operator, how do I know which option to use when I'm searching for a specific. Piece of information, 

Michael: [00:32:20] yeah. I think that's pretty much dictated by a personal preferences and also how that data needs to be stored and exploded later on.

If I'm an operator and I just need to do a quick search, I can go into the UI, right? I can, I could go into the UI type decrees, see the results. That's fine. But then if I need to. Take that information and send it somewhere else. Then the difficulties start right. Then I need to copy paste it into another file and then open, I dunno, an editor put it in there, change it.

So if you're just inspecting something on your own, on the UI, that's totally fine. I think when NuShell culture is shell scripting comes in is where you really need those more automated workflows and you have more. Data transformation needs, right? Where, again, as we said, take data, transform it in different ways, because that's something you can't do on the UI.

On the UI. You can look at your document, you can run a nickel or something, but then when the challenge is okay, now I need to change this information. I need to maybe store it in XML because, or I need to create an extra file for my, on the lists. That's not something that you I can provide, right? And this is where.

New Zealand culture is she'll come in where I can take that information and create an excellent file out of it. Which I would have to do manually otherwise, which is of course, very tedious and error prone. 

Amy: [00:33:41] Okay. And so why did you choose new sh why did Couchbase specifically choose NuShell to integrate with, instead of going for a bash let's say, or PowerShell or something?

Michael: [00:33:53] So both. Okay. So bash and PowerShell. They are, the only thing we could do is basically write tools on top. The great thing with Neushul is that again, it's written in rust and it's written in a way that we can extend it. So basically what we do is we take the NuShell source code and this is where rust and its creative ecosystem comes in.

So NuShell is distributed not only as a binary, but also as a library as well, we call it, create in Russ, which is like a package did you can use in your  manager included, and then we are adding our own commands on top. And build a new binary on top, right? So we are basically giving users a binary they can run.

And when they started, they get everything of NuShell and coach based off from top. 

Amy: [00:34:43] This is something that binary on top of like a PowerShell. Yeah, 

Michael: [00:34:46] exactly. I don't know if that's even possible, but if it is, I wouldn't ride, I want to ride CNC plus to be honest and the nice thing here really is because Neushul was designed with extensibility in mind and the safe aspects of, for us, it really gives me confidence that when I extend NuShell, I can create a binary that is stable, that doesn't seek for it.

Basically that's something I can give users with trust and. It just works. And this is an aspect, especially that, usually uses put a lot of things in their application stack, but the database is something that users need to trust, right? Because there are a couple scenes that users don't forgive databases.

One of which is data loss, for example, right? A database is not allowed to lose your data. And also the commands or the binaries we provide. Users have a very high expectation of the stability and functionality and using building a cup of NuShell, putting a top of rust really gives us that safety, but also we don't have to reinvent the wheel, so we can build on this great foundation of NuShell. Give the users the flexibility. Add our pieces on top. And it's at Couchbase. There is currently basically so two people working on it, but on the sides, right? It's not something like two people full time. And because of all this work that Jonathan and all the other great open source contributors are putting in to NuShell, we can benefit from that.

All the new features that are rolling in week after week, we just pick those up super happy to see them coming in, put our stuff on top of which of course we also try to improve, but the end result is so much more than the individual pieces. Basically. It's just it's really a tool that is fun to use.

And the other aspect is we're really trying to push it internally as well. Cultural basis of picking it up internally, trying it for the use cases and giving us feedback, Hey, we need that new command or that thing doesn't really work or how can we extend it? And in my opinion, again, coming back, going full circle back to the original discussion.

Having a tool that is fun to use, right? Like usually database is another associate with fun, right? Like it's a place, you store your data and that's it. But if you can give a developer operations, the tool, they're really like to open them up, play with it and see the results they want. And then basically go away smiling.

That's the end goal I want to achieve. And Neushul really gives us this big leap forward, basically, which we can attach to. 

Amy: [00:37:06] JT. How does it feel as the co-creator to hear such good things about NuShell? 

JT: [00:37:13] I love it. I love it. I feel like you get these opportunities where you're like if we make an extensible, maybe someone will come along and plug their ideas into that.

And. Whether that's a new contributor coming along and adding a new command or whether it's like Michael and Couchbase and this idea of taking NuShell and then growing a whole shell, like starting NuShell and then building on top of it. So you have a new experience added to it. I think it's awesome to be able to watch it grow in that way.

Amy: [00:37:45] Have any of your other customers done something similar? 

JT: [00:37:48] That's a good question. There are other people using new shot all in different contexts. So I think Couchbase is the only one I know of. That's like trying to build a full NuShell around that, but you can take a, as Michael said, like we're built on these libraries or crates and Russ speak.

And so you can use some piece of that in other contexts. So people are experimenting with using parts of say a NuShells. Data processing system in one context or a NuShells way that we do structured data in another context and yeah, to really see the system get pulled in those directions. I kinda think is fun because that gives a lot of space for different people from different backgrounds to jump in and really grow NuShell in their kind of their experience circles.

And really what it leaves is a much stronger NuShell. So I'm all for it. 

Amy: [00:38:41] Yeah. Amazing. So JT and Michael, thank you so much for coming on the podcast. If we want to find NuShell and UGT online, where can we look? 

JT: [00:38:52] So NuShell, you can go to NuShell.sh, which is the website. We've got to get hub repo.

We've got a really active discord that's linked to from the get hub. So definitely check those out.  You can find me at Twitter at J N T R N R.

Amy: [00:39:08] Amazing. Okay, great. And Michael, where can we find you and what you're working on at Couchbase. 

Michael: [00:39:15] Yeah. If you really want to get started with coach Michelle, you just go to culture, pistol, S H which shows you everything you have, see that documentation, download links, everything to check out. If you're using it and you enjoy it, or you don't get to a certain parts on Kita, please fight issues.

We are really at that stage where we are actively looking for feedback and we really want to. Move fast and incorporate it. And if you want to contact me directly, you can do that on Twitter as well. So that's D a S C H L Tasha on Twitter. And I'm pretty active there too. If you pick me up happy to report, 

Amy: [00:39:48] amazing.

I will put all of those links in the show note. Thank you so much, everybody. 

Michael: [00:39:53] Thank you very much for having us. 

Amy: [00:39:54] If you like this episode, don't forget to like share and subscribe to our hacker nude podcast channels. And as always, you can find us at hacker noon online on Instagram, Twitter, and LinkedIn at hacker noon.

And if you didn't like this episode, you can tweet JT about it. I don't care. Goodbye.