Runchise

runchise

SOARING

Data on more than 800 schools shows big disparities in Florida In the 15 years he’s been teaching, high school literature teacher Jude Mirambeau has never faced a school year like the past one. In March, as the COVID-19 pandemic spread, his state of Florida, which is home to some of the largest school districts in the nation, shuttered its schools in an attempt to slow infections. Teachers shifted to “distance learning.” Mirambeau’s district, the Broward County school district, which has about 270,000 students, loaned out tens of thousands of computers to help get students access. He moved his lessons online, using educational software to assign his students work about Arthur Miller’s “The Crucible,” and did his best to stay in contact and field questions—but it was hard. “I can’t see the student in front of me. I can’t provide accommodations for those who are disabled. I can’t help those who are English-language learners,” said Mirambeau, who teaches 11th-graders at McArthur High School in Hollywood, Fla. "I can’t see the student in front of me. I can’t provide accommodations for those who are disabled. I can’t help those who are English-language learners". Jude Mirambeau, Broward County teacher Mirambeau and his colleagues at the school, where over 70 percent of the student body is considered economically disadvantaged, started noticing a problem: Some students weren’t participating in online learning. Some students suffered technical problems with their issued laptops, Mireambeau said. For others, this was the first computer in their homes, so they shared it with family members. And there were kids who never signed on to the learning software at all. But teachers in other parts of the county told him they weren’t having the same issues. “Some schools out west didn’t need the technology—more of your wealthier areas, more affluent,” Mirambeau said. backs up Mirambeau’s observations. The Markup analyzed data from hundreds of Broward County schools from a two-week period in April: Overwhelmingly, The Markup found, schools in Broward County with higher percentages of minority and/or low-income students and schools already struggling with academic achievement fared the worst as classes moved online. In many cases, whiter, wealthier schools with higher achievement scores actually improved attendance after the switch. Online learning data obtained by The Markup Attendance at McArthur High, for example, dipped to 88.2 percent during two weeks of distance learning in April, from an average of 91.7 percent last school year. Meanwhile, Cypress Bay High School, a low-poverty school in a western suburb in Broward County, saw its attendance remain relatively stable—95.4 percent compared to an average of 95.7 percent in the previous school year. While attendance says little about how much kids are actually learning in school, it is a basic indicator of engagement—and in this case, it shows how already disadvantaged kids in the nation’s sixth largest school district didn’t, or couldn’t, switch to remote classes. The effects, for both Florida and the United States as a whole, could be severe: Kids from poorer schools are in danger of falling even farther behind their peers, and graduation rates might dip, for an education and achievement gap for a whole generation. setting the stage “Look, what we’re going to see, we know, from all the data, that children who are already academically behind are going to experience the greater learning losses,” says Maria Scruggs, who, as head of the St. Petersburg branch of the NAACP, raised concerns with Pinella County school officials about learning gaps back in April. The NAACP in Florida in recent years has focused heavily on educational inequities in the state, which were before the pandemic. A is running a tutoring program for kids adjusting to online learning. already significant branch of the organization Scruggs has since left her post to run for a seat on the county commission. She questioned whether enough is being done to keep students, particularly low-income students who might not have the same access to computers and tutors, tuned in to instruction. “We know that our children are going to have suffered learning losses,” she said. “We know that.” With school starting again this month, she said, it’s not clear Florida districts will be able to improve engagement for low-income students. Schools Had Wildly Different Attendance Rates for Online Learning In contrast to many districts around the country, most of which stopped taking attendance after the switch to distance learning, Florida tracked log-in and digital participation rates during the pandemic. The exact units of measurement varied, but generally districts tallied the percentage of students who interacted with e-learning software during the school week. The Markup asked the 10 largest counties in the state for the data and obtained usably detailed participation and attendance statistics from more than 800 schools in three districts—Broward, Miami-Dade, and Hillsborough—through public records requests. Broward provided the most detailed data and was the only district to provide daily attendance and enrollment counts for each school. We compared the average remote attendance for each school in Broward County to attendance rates during the previous school year. We also looked at each school’s demographic information and the letter grades Florida gives to schools in evaluating academic achievement. Often, schools with a high number of low-income students also had a high number of minority students—a described double segregation, or segregation by both race and poverty, as “a strikingly notable trend in Florida” and noted that Black and Latino students “are far more likely to be segregated in schools with low achieving students.” 2017 Florida State University study The Markup’s analysis found that, overwhelmingly, those already struggling schools fared the worst, seeing the largest drops in attendance after the switch to online learning. Schools that were already on improvement plans because of low academic achievement had the biggest drops. And —which in Florida are places where students with behavioral or academic problems —consistently showed the lowest participation rates. alternative schools might be sent One alternative school, Dave Thomas Education Center, lost 42 students over the course of two weeks, according to enrollment numbers—the equivalent of 7.6 percent of the original student body. In an interview, Daniel Gohl, Broward County’s chief academic officer, did not dispute The Markup’s analysis but said the county’s own data did not find a significant correlation between a school’s historical academic achievement and online participation. The county did not share or detail its analysis. Gohl did say that alternative schools, where students may have unique disability or behavioral issues, presented new challenges during the switch to remote learning and that there was a notable link between those schools and lower attendance. “We have a responsibility,” he said of alternative schools. “All school districts have a responsibility to having those needs addressed.” Schools in Other Parts of the State Also Showed Stark Disparities The Markup also looked at schools in other Florida counties and found wide gaps in attendance. Every single one of the 2,414 students at in Tampa participated in schooling during the week of April 10, according to the district’s count. But just six miles away, at Gaither High School, which has a lower student achievement and where the rate of economically disadvantaged students is about 20 points higher, the situation was very different. That week at Gaither, 922 students never interacted with the distance learning system—45 percent of the entire student body. Sickles High School grade 100% Online participation rate of students at Hillsborough County's Sickles High School during the week of April 10. “We leveraged teachers, administrators, school counselors, social workers, and school psychologists to reach out to students who had not engaged in eLearning,” Tanya Arja, a spokesperson for Hillsborough County Public Schools, said in an emailed statement to The Markup. The district also lent 40,000 laptops to families and provided about 1,500 Wi-Fi hotspots to give students internet access. Arja said 98 percent of students in the district connected with teachers through the remote process. But data shows the students who didn’t participate weren’t evenly distributed. At three charter schools in the district, less than 50 percent of students participated some weeks, even as other schools hit nearly 100 percent attendance. 55% Online participation rate of students at Hillsborough County's Gaither High School during the week of April 10. In Miami-Dade County, some days only half the kids at some schools were counted for remote learning in April—while other schools saw 100 percent attendance. On average, in Miami-Dade County, according to data analyzed by The Markup, high-poverty schools showed a nearly 10 percent drop in attendance, while the wealthiest  schools experienced a less than one percent drop. A spokesperson for the Miami-Dade County school district, Natalia Zea, said in an emailed statement that the district tracked statistics on subgroups of students, including English-language learners and the “economically disadvantaged.” Zea said “student attendance by subgroup within the district as a whole was similar to that of students overall” but that “additional analysis on the performance of students within subgroups has not been completed.” The Pandemic Didn’t Start Educational Disparities—but It Has Made Them Worse Marie Generazio of Coral Springs, Fla., has two children in Broward County public schools. One will be heading into fourth grade in the fall, and the other into first. Generazio, meanwhile, has been taking online college classes to become a paralegal. She’s been taking care of her kids and occasionally letting them use her computer, if their school-issued laptops aren’t working. But keeping up with the kids’ assignments, especially if they have strict time requirements, isn’t always easy. The past year, she said, has been trying, as the family does “lots of shuffling” to make it all work. They didn’t always meet time requirements for assignments and log-in deadlines for attendance. “You have two kids here that you have to play teacher for,” she said. Phyllis Jordan, editorial director at FutureEd, an education think tank at Georgetown University, said lots of families are having a similar experience. Students may not have the equipment at home to log in to class, she said, or their parents might be essential workers, unable to watch them during the day to ensure they sign in for school. It’s often “the same populations” of students who had trouble before the pandemic hit, she said, frequently those from lower-income families. The pandemic didn’t create educational gaps, she said, but “it definitely has exacerbated the problem.” And those gaps are becoming even harder to measure. Broward County’s detailed records are an exception, not the norm, nationwide. “If we’re not tracking their attendance,” superintendent of the Broward County school district Robert Runcie recently , “then in some ways we are neglecting our responsibility to make sure those kids are safe.” He told the publication that the district had sent Wi-Fi hot spots to some students after noticing they hadn’t logged in to the system. told Reuters The New School Year and Continuing Repercussions In June, state officials presented a plan for schools to re-open with in-person instruction in the fall, citing both ongoing achievement gaps in education and damage to the wider economy, . They’ve since given leeway for schools to begin the year with remote instruction. according to the Tampa Bay Times “Getting back to the school year is going to be really, really important to the well-being of our kids,” the governor said, according to the outlet. Broward County is set to resume online classes this month. In response to the state’s push for schools to resume in-person learning, the , a statewide federation of teachers’ and education workers’ unions, has asking a judge to block the return of in-person schooling until additional protective measures are put in place. The NAACP also joined the suit, saying as difficult as online learning is, schools should open only once it’s safe. Florida Education Association filed a lawsuit However Florida schools reopen this month, in an attempt to contain the immediate economic fallout of low attendance due to remote learning or pandemic-related factors, the state has also temporarily untethered funding from attendance for the year and is instead from past attendance. Other have taken similar measures. relying on projections states With distance learning staying around for the foreseeable future, districts need to make sure students aren’t left behind, said Hedy Chang, director of Attendance Works, a group dedicated to improving attendance. “You have redefined, ‘What is school?’ ” Chang said. “And what having equitable access to learning resources looks like.” Credits Originally published as " " with the Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) license. Remote Learning During the Pandemic Has Hit Vulnerable Students the Hardest

Already Struggling with School? The Pandemic Made it Worse. 

How To Make Your Codebase More Readable

Philip K. Dick, the Matrix,  Aliens and Alternate History: My Conversation with Tessa B. Dick

A , we addressed some important steps to advance past the “beginner” stage of Haskell. We learned how to organize your project and how to find the relevant documentation. This week we’re going to continue to look at another place where we can make a big step up. We’ll explore how to expand our vocabulary on monad usage. few weeks ago Monads are a vital component of Haskell. You can’t use a lot of libraries unless you know how to incorporate their monadic functions. These functions often involve a monad that is custom to that library. When you’re first starting out, it can be hard to know how to incorporate these monads into the rest of your program. In this article, we’ll focus on a specific pattern a lot of monads and libraries use. I call this pattern the “run” pattern. Often, you’ll use a function with a name like or , where is the name of the monad. These functions will always take a monadic expression as their first argument. Then they'll also take some other initialization information, and finally return some output. This output can either be in pure form or a different monad you’re already using like . We’ll start by seeing how this works with the monad, and then move onto some other libraries. runXXX runXXXT XXX IO State Once you grasp this topic, it seems very simple. But a lot of us first learned monads with a bad mental model. For instance, the first thing I learned about monads was that they had side effects. And thus, you can only call them from places that have the same side effects. This applies to IO but doesn’t generalize to other monads. So even though it seems obvious now, I struggled to learn this idea at first. But let’s start looking at some examples of this pattern. For a more in depth look at monads, check out our series on ! We start by learning about simpler things like functors. Then we eventually work our way up to monads and even monad transformers! Functional Data Structures The Basics of “Run”: The State Monad Let’s start by recalling the monad. This monad has a single type parameter, and we can access this type as a global read/write state. Here’s an example function written in the monad: State State stateExample :: State Int (Int, Int, Int)stateExample = do  a <- get  modify (+1)  b <- get  put 5  c <- get  return (a, b, c) If this function is confusing, you should take a look at the . It’ll at least show you the relevant type signatures. First we read the initial state. Then we modify it with some function. Finally we completely change it. documentation for [State](http://hackage.haskell.org/package/mtl-2.2.2/docs/Control-Monad-State-Lazy.html) In the example above, if our initial state is 1, we’ll return as the result. If the initial state is 2, we’ll return . But suppose we have a pure function. How do we call our state function? (1,2,5) (2,3,5) pureFunction :: Int -> IntpureFunction = ??? The answer is the function. We can check the documentation and find its type: runState runState :: State s a -> s -> (a, s) This function has two parameters. The first is a action. We’ll pass our function above as this parameter! Then the second is the initial state, and this is how we’ll configure it. Then the result is pure. It contains our result, as well as the final value of the state. So here’s a sample call we can make that gives us this monadic expression in our pure function. We’ll call it from a clause, and discard the final state: State where pureFunction :: Int -> IntpureFunction input = a + b + c  where    ((a,b,c), _) = runState stateExample input This is the simplest example of how we can use the pattern. runXXX Upgrading to Transformers Now, suppose our function isn’t quite pure. It now wants to print some of its output, so it’ll need the monad. This means it’ll use the monad transformer over : State IO StateT IO stateTExample :: StateT Int IO (Int, Int, Int)stateTExample = do  a <- get  lift $ print “Initial Value:”  lift $ print a  modify (+1)  b <- get  lift $ putStrLn “After adding 1:”  lift $ print b  put 5  c <- get  lift $ putStrLn “After setting as 5:”  lift $ print c  return (a, b, c) Now instead of calling this function from a pure format, we’ll need to call it from an function. But once again, we’ll use a function. Now though, since we’re using a monad transformer, we won’t get a pure result. Instead, we’ll get our result in the underlying monad. This means we can call this function from . So let’s examine the type of the function. We’ve substituted for the generic monad parameter : IO runXXX IO runStateT IO m runStateT :: StateT s IO a -> s -> IO (a, s) It looks a lot like , except for the extra parameters! Instead of returning a pure tuple for the result, it returns an action containing that result. Thus we can call it from the monad. runState IO IO IO main :: IO ()main = do  putStrLn “Please enter a number.”  input <- read <$> getLine  results <- runStateT stateTExample input  print results We’ll get the following output as a result: Please enter a number.10Initial Value:10After adding 111After setting as 55(10, 11, 5) Using “run” for Libraries This pattern will often extend into libraries you use. For example, in our , we examine the library. A lot of the individual parser combinators in that library exist in the or monad. So we can combine a bunch of different parsers together into one function. series on parsing Megaparsec Parsec ParsecT But then to run that function from your normal code (or another monad), you need to use the function. Let’s look at its type signature: IO runParserT runParserT  :: Monad m  -> ParsecT e s m a  -> String -- Name of source file  -> s -- Input for parser  -> m (Either (ParseError (Token s) e) a) There are a lot of type parameters there that you don’t need to understand. But the structure is the same. The first parameter to our function is the monadic action. Then we’ll supply some other inputs we need. Then we get some result, wrapped in an outer monad (such as ). run IO We can see the same pattern if we use the library to make client-side API calls. Any call you make to your API will be in the monad. Now here’s the type signature of the function: servant-client ClientM runClientM runClientM :: ClientM a -> ClientEnv -> IO (Either ServantError a) So again, the same pattern emerges. We’ll compose our monadic action and pass that as the first parameter. Then we’ll provide some initial state, in this case a . Finally, we’ll get our result ( ) wrapped in an outer monad ( ). ClientEnv Either ServantError a IO Monads within Expressions It’s also important to remember that a lot of basic monads work without even needing a function! For instance, you can use a or monad to take out some of your error handling logic: runXXX Maybe Either divideIfEven :: Int -> Maybe IntdivideIfEven x = if x `mod` 2 == 0  then Just (x `quot` 2)  else Nothing

dividesBy8 :: Int -> BooldividesBy8 = case thirdResult of  Just _ -> True  Nothing -> False  where    thirdResult :: Maybe Int    thirdResult = do      y <- divideIfEven x      z <- divideIfEven y      divideIfEven z Conclusion Monads are the key to using a lot of different Haskell libraries. But when you’re first starting out, it can be very confusing how you call into these functions from your code. The same applies with some common monad transformers like and . The most common pattern to look out for is the pattern. Master this pattern and you’re well on your to understanding monads and writing better Haskell! Reader State runXXXT For a closer look at monads and similar structures, make sure to read our series on . If the code in this article was confusing, you should definitely check it out! And if you’ve never written Haskell but want to start, download our ! Functional Data Structures Beginners Checklist

Making the Jump II: Using More Monads

A couple weeks ago, we saw in conjunction with Heroku to deploy our Haskell application. The process resulted in a simpler Circle CI config than we had before, as we let Docker do most of the heavy lifting. In particular, we no longer needed to download and build stack ourselves. We specified the build process in our Dockerfile, and then called . We also saw a couple different ways to login to these services from our Circle CI box. how to use Docker docker build In the future, we’ll look at ways to use more diverse deployment platforms than Heroku. In particular, we’ll look at AWS. But that’s a tough nut to crack, so it might be worthy of its own series! For now, we’ll conclude our series on deployment by looking at the Github developer API. Most projects you’ll work on use Github for version control. But with the API, there are a lot of interesting tricks that can make your experience cooler! This week, we’ll see how to setup a server that will respond to events that happen within our repository. Then we’ll see how we can send our own events from the server! You can follow along with this code by looking at ! this Github repository This article builds a lot on our knowledge of the Servant library. If you’ve never used that before, I highly recommend you read our series. You’ll learn about Servant and much more! You can also download our for more tools to use in your applications. Haskell Web Skills Production Checklist Github Webhooks Primer First let’s understand the concept of webhooks. Many services besides Github also use them. A webhook is an integration where a service will send an HTTP request to an endpoint of your choosing whenever some event happens. Webhooks are often a way for you to get some more advanced functionality out of a system. They can let you automate a lot of your processes. With Github, we can customize the events where this occurs. So for instance, we can trigger a request whenever creates a pull request. In this article, we’ll set up a very simple server that will do just that. When they open a new PR, we’ll add a comment saying we’ll take a look at the pull request soon. We’ll also have the comment tag our account so we get a notification. The Github part of this is easy. We go to the settings for our repository, and then find the “Webhooks” section. We’ll add a webhook for custom events, and we’ll only check the box next to “Pull Requests”. We’ll assign this to the URL of a Server that we’ll put up on a Heroku server, hitting the endpoint. /api/hook Building our Server First let’s make a data type for a Github request. This will be a simple two-constructor type. Our first constructor will contain information about an opened pull request. We’ll want to get the user’s name out of the request object, as well as the URL for us to send our comment to. We’ll also have an constructor for when the request isn’t about an open pull request. Other data GithubRequest =  GithubOpenPRRequest Text Text | -- User’s name, comments URL  GithubOtherRequest  deriving (Show) So we need a simple server that listens for requests on a particular endpoint. As we have in the past, we’ll use Servant for this process. Our endpoint type will use our desired path. Then it will also take a request body with our GithubRequest. We’ll listen for a post request, and then return a as our result, to help debug. Text type ServerAPI = “api” :> “hook” :>   ReqBody ‘[JSON] GithubRequest :> Post ‘[JSON] Text Now we need to specify a instance for our request type. Using the , we’ll find a few fields we need to read to make this happen. First, we’ll check that, indeed, this request has a pull request section and that it’s action is “opened”. If these aren’t there, we’ll return : FromJSON documentation Other instance FromJSON GithubRequest where  parseJSON = withObject “GithubRequest” $ \o -> do    (action :: Maybe Text) <- o .:? “action”    prSectionMaybe <- o .:? “Pull_request”    case (action, prSectionMaybe) of      (Just “opened”, Just pr_section :: Maybe Value) -> do        …      _ -> return GithubOtherRequest Now we can fetch the user section and the comments URL from the section. We do this with a function on a object like so: pull_request Data.Aeson where  fetchUserAndComments o’ = do    uSection <- o’ .: “user”    commentsURL <- o’ .: “comments_url”    return (uSection, commentsURL) Note we want , NOT ! We want to leave a single comment, rather than performing a full review of this PR. It was VERY annoying to figure out that the documentation covers this under the section, NOT the section on pull requests! Once we get the user section and comments, URL, we need one more step. We’ll get the user name out of the section, and we’ll return our final request! comments_url review_comments_url Issues instance FromJSON GithubRequest where  parseJSON = withObject “GithubRequest” $ \o -> do    (action :: Maybe Text) <- o .:? “action”    prSectionMaybe <- o .:? “Pull_request”    case (action, prSectionMaybe) of      (Just “opened”, Just pr_section :: Maybe Value) -> do        (userSection :: Value, commentsURL :: Text) <-          withObject “PR Section” fetchUserAndComments prSection        userName <-          withObject “User Section” (\o’ -> o’ .: “login”) userSection        return $ GithubOpenPRRequest userName commentsURL      _ -> return GithubOtherRequest Handling the Endpoint Now we need a handler function for endpoint. This handler will pattern match on the type of request and return a debugging string. If we have indeed found a request to open the PR, we’ll also want to call another function that will add our comment: IO hookHandler :: GithubRequest -> Handler TexthookHandler GithubOtherRequest =  return “Found a non-PR opening request.”hookHandler (GithubOpenPRRequest userName commentsURL) = do  liftIO $ addComment userName commentsURL  return $ “User: “ <> userName <>     “ opened a pull request with comments at: “ <> commentsURL

addComment :: Text -> Text -> IO ()... Adding a Comment In order to add a comment to this pull request, we’ll need to hit the Github API with our own request. Again, we’ll do this using Servant’s magic! First, let’s make another API type to represent Github’s own developer API. Since we’re getting the full comments URL as part of our request, we don’t need any path components here. But we will need to authenticate using : BasicAuth type GithubAPI = BasicAuth “GithubUser” () :>  ReqBody GitPRComment :> Post ‘[JSON] () Our will only need a for the body of the comment. So let’s make a simple wrapper and add a instance for it: GitPRComment Text newtype ToJSON newtype GitPRComment = GitPRComment Text

instance ToJSON GitPRComment where  toJSON (GitPRComment body) = object [ “body” .= body ] We can create a client function for this API now using the magic function from : client Servant.Client sendCommentClient :: BasicAuthData -> GitPRComment -> ClientM ()sendCommentClient = client (Proxy :: Proxy GithubAPI) Now to build our commenting function, we’ll start by building the auth data. import qualified Data.ByteString.Char8 as BSC

...addComment :: Text -> Text -> IO ()addComment userName commentsURL = do  gitUsername <- getEnv “GITHUB_USERNAME”  gitPassword <- getEnv “GITHUB_PASSWORD”  let authData = BasicAuthData (BSC.pack gitUsername)                               (BSC.pack gitPassword)  ... Now we’ll set up our client environment using the comments URL: addComment :: Text -> Text -> IO ()addComment userName commentsURL = do  ...  manager <- newManager tlsManagerSettings  baseUrl <- parseBaseUrl (Data.Text.unpack commentsURL)  let clientEnv = clientEnv maanger baseUrl  ... We’ll add a simple function taking our admin’s username and composing the body of the comment. We’ll tag ourselves as well as the user who opened the PR: addComment :: Text -> Text -> IO ()addComment userName commentsURL = do  …  where     commentBody adminName = GitPRComment $       “Thanks for posting this @” <> userName <>       “! I’ll take a look soon! - @” <> adminName Now we wrap everything together by making our client call. And that’s it! addComment :: Text -> Text -> IO ()addComment userName commentsURL = do  gitUsername <- getEnv “GITHUB_USERNAME”  gitPassword <- getEnv “GITHUB_PASSWORD”  let authData = BasicAuthData (BSC.pack gitUsername)                               (BSC.pack gitPassword)  manager <- newManager tlsManagerSettings  baseUrl <- parseBaseUrl (Data.Text.unpack commentsURL)  let clientEnv = clientEnv maanger baseUrl  runClientM (sendCommentClient                authData               (commentBody gitUsername))             clientEnv  return ()  where    commentBody = ... Conclusion Services like Github do their best to provide a good user experience to all their normal users. But if you get a little bit advanced, you can often customize their behavior to a great degree! Notice how important it is to know how to setup a simple server. This gives you limitless freedom to manipulate the system and add your own behaviors. It’s a cool perk of learning these specific web skills. If you want to see the full code I wrote for this article, check it out on ! this Github repo To learn about more web skills that can magnify your programming ability, check out our . It’ll walk you through some different Haskell libraries, like Persistent for databases, and Servant for web servers. You can also download our . It’ll give you a lot more ideas of libraries to use to enhance your Haskell experience! Haskell Web Skills Series Production Checklist

Advanced Github: Webhooks and Automation

As you might know, concurrency and parallelism are different terms. Concurrency means that an application is making progress on more than one task at the same time (concurrently); and parallelism means that an application splits its tasks up into smaller subtasks which can be processed in parallel, for instance on multiple CPUs at the exact same time. Let’s imagine a simple example where there is an application having progress on multiple tasks, and applying concurrency would help a lot:A university began to use a new database as a source of questions for their tests. Because this database has way more questions than the other, they are having problems while checking if the answers are right in order to calculate the final grade for each student. Since it’s too much data, it makes no sense to cache and impossible to keep on memory. Well, the performance got low because they had to get all answers and check the right choice for each answer before calculating the final grade, making the scripts consume a lot of time to run.This problem could be easily solved by using concurrency. Applying it in one task would make it have progress without having to wait for the others. Then, while one task is searching answers on DB the other is already checking if the answer is right, and another one can be aggregating the answers to persist back the data. :git clone :shodocan/golang-blocking-channels-example.git Now, let’s implement a great solution for this using Golang git@gitlab.com On this example, we will be using Datastore (from Google Console Platform), but you can use any database with cursor support. To generate your data:_cd utilsexport PROJECT_ID={GOOGLE_PROJECT_ID}go run utils.go_ First, check the script in the directory without-blocking-channels. It took 882 seconds to check 3000 answers of 300 students. Now check the directory . Just by changing it to pipelines reduced that number to 849. Ok, no big deal yet but using pipelines we can make huge changes on steps that make the code slow. using-blocking-channels Check your main task and replace it to: So, now you are running 10 calcGrade and 2 upgradeGrades. Those tasks are slow because they need to read or write the DB. However, just because of this change, now our script takes 230 seconds to process all 3000 answers. You can go deeper and check every step to check what is slowing you down and make it faster. Great, right? Now, let’s understand the go tools used and how they work. : I think it’s the most powerful weapon that golang provides for concurrent tasks. Go blocking channels works by letting your tasks communicate with each other in a clean and elegant way. It’s an only one-way communication where the producer get blocked to publish more messages until the consumer can process it. Allowing you to don’t waste your time reading data from a database while the task what process it can keep the reader speed leading our software to consume unneeded resources. Blocking Channels Example:Check After checking if an answer is right or wrong, we accumulate the grades in order to make fewer requests to update the students grades on the database. grades.go. When using channels, we should always close the channel, so the consumer can know its time to stop listening. : Sometimes you read packs of data from the database to make fewer requests. In this case, it makes no sense to block my producer if it already has a pack of data on the hand. We can use the buffer to let him publish the pack of data and start to get the new data moments before the consumer consumes the actual data. Blocking Channels Buffer : it is a tool that allows you to run multiple async tasks and wait for their execution. I used it to merge channels, assuring that the aggregated channel will only close when all the input channels are closed. It’s not impossible to do, but golang made it so easy and elegant to use that I really had to show it. Wait Group I don’t even need to talk about the advantages, the example clearly shows it. Golang made it easier and let the code understandable, testable and elegant.

Concurrency: how it can help you and how you can use GOLANG to reach it easily

In the last four articles or so, we’ve done a real whirlwind tour of Haskell libraries. We created a database schema using Persistent and used it to write basic SQL queries in a type-safe way. We saw how to expose this database via an API with Servant. We also went ahead and added some caching to that server with Redis. Finally, we wrote some basic tests around the behavior of this API. By using Docker, we made those tests reproducible. In this article, we’re going to review this whole process by adding another type to our schema. We’ll write some new endpoints for an type, and link this type to our existing type with a foreign key. Then we’ll learn one more library: . Esqueleto improves on Persistent by allowing us to write type-safe SQL joins. Article User Esqueleto As with the previous articles, there’s a specific branch on the for this series. Go there and take a look at the to see the complete code for this article. Github repository esqueleto branch Adding Article to our Schema So our first step is to extend our schema with our type. We’re going to give each article a title, some body text, and a timestamp for its publishing time. One new feature we’ll see is that we’ll add a foreign key referencing the user who wrote the article. Here’s what it looks like within our schema: Article PTH.share [PTH.mkPersist PTH.sqlSettings, PTH.mkMigrate "migrateAll"] [PTH.persistLowerCase| User sql=users   ...

 Article sql=articles   title Text   body Text   publishedTime UTCTime   authorId UserId   UniqueTitle title   deriving Show Read Eq|] We can use as a type in our schema. This will create a foreign key column when we create the table in our database. In practice, our Article type will look like this when we use it in Haskell: UserId data Article = Article { articleTitle :: Text , articleBody :: Text , articlePublishedTime :: UTCTime , articleAuthorId :: Key User } This means it doesn’t reference the entire user. Instead, it contains the SQL key of that user. Since we’ll be adding the article to our API, we need to add ToJSON and FromJSON instances as well. These are pretty basic as well, so you can check them out if you’re curious. If you’re curious about JSON instances in general, take a look . here at this article Adding Endpoints Now we’re going to extend our API to expose certain information about these articles. First, we’ll write a couple basic endpoints for creating an article and then fetching it by its ID: type FullAPI =       "users" :> Capture "userid" Int64 :> Get '[JSON] User :<|> "users" :> ReqBody '[JSON] User :> Post '[JSON] Int64 :<|> "articles" :> Capture "articleid" Int64 :> Get '[JSON] Article :<|> "articles" :> ReqBody '[JSON] Article :> Post '[JSON] Int64 Now, we’ll write a couple special endpoints. The first will take a User ID as a key and then it will provide all the different articles the user has written. We’ll do this endpoint as . /articles/author/:authorid ... :<|> "articles" :> "author" :> Capture "authorid" Int64 :> Get '[JSON] [Entity Article] Our last endpoint will fetch the most recent articles, up to a limit of 10. This will take no parameters and live at the route. It will return tuples of users and their articles, both as entities. /articles/recent … :<|> "articles" :> "recent" :> Get '[JSON] [(Entity User, Entity Article)] Adding Queries (with Esqueleto!) Before we can actually implement these endpoints, we’ll need to write the basic queries for them. For creating an article, we use the standard Persistent function: insert createArticlePG :: PGInfo -> Article -> IO Int64createArticlePG connString article = fromSqlKey <$> runAction connString (insert article) We could do the same for the basic fetch endpoint. But we’ll write this basic query using Esqueleto in the interest of beginning to learn the syntax. With Persistent, we used list parameters to specify different filters and SQL operations. Esqueleto instead uses a special monad to compose the different type of query. The general format of an esqueleto select call will look like this: fetchArticlePG :: PGInfo -> Int64 -> IO (Maybe Article)fetchArticlePG connString aid = runAction connString selectAction where   selectAction :: SqlPersistT (LoggingT IO) (Maybe Article)   selectAction = select . from $ \articles -> do     ... We use and then provide a function that takes a table variable. Our first queries will only refer to a single table, but we'll see a join later. To complete the function, we’ll provide the monadic action that will incorporate the different parts of our query. select . from The most basic filtering function we can call from within this monad is . This allows us to provide a condition on the query, much as we could with the filter list from Persistent. where_ selectAction :: SqlPersistT (LoggingT IO) (Maybe Article)   selectAction = select . from $ \articles -> do     where_ (articles ^. ArticleId ==. val (toSqlKey aid)) First, we use the lens to specify which value of our table we’re filtering. Then we specify the value to compare against. We not only need to lift our into an , but we also need to lift that value using the function. ArticleId Int64 SqlKey val But now that we’ve added this condition, all we need to do is return the table variable. Now, returns our results in a list. But since we’re searching by ID, we only expect one result. We’ll use so we only return the head element if it exists. We’ll also use once again to unwrap the article from its entity. select listToMaybe entityVal selectAction :: SqlPersistT (LoggingT IO) (Maybe Article)   selectAction = ((fmap entityVal) . listToMaybe) <$> (select . from $ \articles -> do     where_ (articles ^. ArticleId ==. val (toSqlKey aid))     return articles) Now we should know enough that we can write out the next query. It will fetch all the articles that have written by a particular user. We’ll still be querying on the table. But now instead checking the article ID, we’ll make sure the is equal to a certain value. Once again, we’ll lift our user key into an and then again with to compare it in “SQL-land”. articles ArticleAuthorId Int64 SqlKey val fetchArticleByAuthorPG :: PGInfo -> Int64 -> IO [Entity Article]fetchArticleByAuthorPG connString uid = runAction connString fetchAction where   fetchAction :: SqlPersistT (LoggingT IO) [Entity Article]   fetchAction = select . from $ \articles -> do     where_ (articles ^. ArticleAuthorId ==. val (toSqlKey uid))     return articles And that’s the full query! We want a list of entities this time, so we’ve taken out and . listToMaybe entityVal Now let’s write the final query, where we’ll find the 10 most recent articles regardless of who wrote them. We’ll include the author along with each article. So we’re returning a list of of these different tuples of entities. This query will involve our first join. Instead of using a single table for this query, we’ll use the constructor to combine our table with the table. InnerJoin users articles fetchRecentArticlesPG :: PGInfo -> IO [(Entity User, Entity Article)]fetchRecentArticlesPG connString = runAction connString fetchAction where   fetchAction :: SqlPersistT (LoggingT IO) [(Entity User, Entity Article)]   fetchAction = select . from $ \(users `InnerJoin` articles) -> do Since we’re joining two tables together, we need to specify what columns we’re joining on. We’ll use the function for that: on fetchAction :: SqlPersistT (LoggingT IO) [(Entity User, Entity Article)]   fetchAction = select . from $ \(users `InnerJoin` articles) -> do     on (users ^. UserId ==. articles ^. ArticleAuthorId) Now we’ll order our articles based on the timestamp of the article using . The newest articles should come first, so we'll use a descending order. Then we limit the number of results with the function. Finally, we’ll return both the users and the articles, and we’re done! orderBy limit fetchAction :: SqlPersistT (LoggingT IO) [(Entity User, Entity Article)]   fetchAction = select . from $ \(users `InnerJoin` articles) -> do     on (users ^. UserId ==. articles ^. ArticleAuthorId)     orderBy [desc (articles ^. ArticlePublishedTime)]     limit 10     return (users, articles) Caching Different Types of Items We won’t go into the details of caching our articles in Redis, but there is one potential issue we want to observe. Currently we’re using a user’s SQL key as their key in our Redis store. So for instance, the string “15” could be such a key. If we try to naively use the same idea for our articles, we’ll have a conflict! Trying to store an article with ID “15” will overwrite the entry containing the User! But the way around this is rather simple. What we would do is that for the user’s key, we would make the string something like . Then for our article, we’ll have its key be . As long as we deserialize it the proper way, this will be fine. users:15 articles:15 Filling in the Server handlers Now that we’ve written our database query functions, it is very simple to fill in our Server handlers. Most of them boil down to following the patterns we’ve already set with our other two endpoints: fetchArticleHandler :: PGInfo -> Int64 -> Handler ArticlefetchArticleHandler pgInfo aid = do maybeArticle <- liftIO $ fetchArticlePG pgInfo aid case maybeArticle of   Just article -> return article   Nothing -> Handler $ (throwE $ err401 { errBody = "Could not find article with that ID" })

createArticleHandler :: PGInfo -> Article -> Handler Int64createArticleHandler pgInfo article = liftIO $ createArticlePG pgInfo article

fetchArticlesByAuthorHandler :: PGInfo -> Int64 -> Handler [Entity Article]fetchArticlesByAuthorHandler pgInfo uid = liftIO $ fetchArticlesByAuthorPG pgInfo uid

fetchRecentArticlesHandler :: PGInfo -> Handler [(Entity User, Entity Article)]fetchRecentArticlesHandler pgInfo = liftIO $ fetchRecentArticlesPG pgInfo Then we’ll complete our like so: Server FullAPI fullAPIServer :: PGInfo -> RedisInfo -> Server FullAPIfullAPIServer pgInfo redisInfo = (fetchUsersHandler pgInfo redisInfo) :<|> (createUserHandler pgInfo) :<|> (fetchArticleHandler pgInfo) :<|> (createArticleHandler pgInfo) :<|> (fetchArticlesByAuthorHandler pgInfo) :<|> (fetchRecentArticlesHandler pgInfo) One interesting thing we can do is that we can compose our API types into different sections. For instance, we could separate our into two parts. First, we could have the type from before, and then we could make a new type for . We can glue these together with the e-plus operator just as we could individual endpoints! FullAPI UsersAPI ArticlesAPI type FullAPI = UsersAPI :<|> ArticlesAPI

type UsersAPI =      "users" :> Capture "userid" Int64 :> Get '[JSON] User :<|> "users" :> ReqBody '[JSON] User :> Post '[JSON] Int64

type ArticlesAPI = "articles" :> Capture "articleid" Int64 :> Get '[JSON] Article :<|> "articles" :> ReqBody '[JSON] Article :> Post '[JSON] Int64 :<|> "articles" :> "author" :> Capture "authorid" Int64 :> Get '[JSON] [Entity Article] :<|> "articles" :> "recent" :> Get '[JSON] [(Entity User, Entity Article)] If we do this, we’ll have to make similar adjustments in other areas combining the endpoints. For example, we would need to update the server handler joining and the client functions. Writing Tests Since we already have some user tests, it would also be good to have a few tests on the Articles section of the API. We’ll add one simple test around creating an article and then fetching it. Then we’ll add one test each for the “articles-by-author” and “recent articles” endpoints. So one of the tricky parts of filling in this section will be that we need to make test object. But we'll need them to be functions on the User ID. This is because we can’t know a priori what SQL IDs we'll get when we insert the users into the database. But we can fill in all the other fields, including the published time. Here’s one example, but we’ll have a total of 18 different “test” articles. Article testArticle1 :: Int64 -> ArticletestArticle1 uid = Article { articleTitle = "First post" , articleBody = "A great description of our first blog post body." , articlePublishedTime = posixSecondsToUTCTime 1498914000 , articleAuthorId = toSqlKey uid }

-- 17 other articles and some test users as well… Our before hooks will create all these different entities in the database. In general, we’ll go straight to the database without calling the API itself. Like with our users tests, we’ll want to delete any database items we create. Let’s write a generic after-hook that will take user IDs and article IDs and delete them from our database: deleteArtifacts :: PGInfo -> RedisInfo -> [Int64] -> [Int64] -> IO ()deleteArtifacts pgInfo redisInfo users articles = do void $ forM articles $ \a -> deleteArticlePG pgInfo a void $ forM users $ \u -> do   deleteUserCache redisInfo u   deleteUserPG pgInfo u It’s important we delete the articles first! If we delete the users first, we’ll encounter foreign key exceptions! Our basic create-and-fetch test looks a lot like the previous user tests. We test the success of the response and that the new article lives in Postgres as we expect. beforeHook4 :: ClientEnv -> PGInfo -> IO (Bool, Bool, Int64, Int64)beforeHook4 clientEnv pgInfo = do userKey <- createUserPG pgInfo testUser2 articleKeyEither <- runClientM (createArticleClient (testArticle1 userKey)) clientEnv case articleKeyEither of   Left _ -> error "DB call failed on spec 4!"   Right articleKey -> do     fetchResult <- runClientM (fetchArticleClient articleKey) clientEnv     let callSucceeds = isRight fetchResult     articleInPG <- isJust <$> fetchArticlePG pgInfo articleKey     return (callSucceeds, articleInPG, userKey, articleKey)

spec4 :: SpecWith (Bool, Bool, Int64, Int64)spec4 = describe "After creating and fetching an article" $ do it "The fetch call should return a result" $ \(succeeds, _, _, _) -> succeeds `shouldBe` True it "The article should be in Postgres" $ \(_, inPG, _, _) -> inPG `shouldBe` True

afterHook4 :: PGInfo -> RedisInfo -> (Bool, Bool, Int64, Int64) -> IO ()afterHook4 pgInfo redisInfo (_, _, uid, aid) = deleteArtifacts pgInfo redisInfo [uid] [aid] Our next test will create two different users and several different articles. We’ll first insert the users and get their keys. Then we can use these keys to create the articles. We create five articles in this test. We assign three to the first user, and two to the second user: beforeHook5 :: ClientEnv -> PGInfo -> IO ([Article], [Article], Int64, Int64, [Int64])beforeHook5 clientEnv pgInfo = do uid1 <- createUserPG pgInfo testUser3 uid2 <- createUserPG pgInfo testUser4 articleIds <- mapM (createArticlePG pgInfo)   [ testArticle2 uid1, testArticle3 uid1, testArticle4 uid1   , testArticle5 uid2, testArticle6 uid2 ] ... Now we want to test that we when call the articles-by-user endpoint, we only get the right articles. We’ll return each group of articles, the user IDs, and the list of article IDs: beforeHook5 :: ClientEnv -> PGInfo -> IO ([Article], [Article], Int64, Int64, [Int64])beforeHook5 clientEnv pgInfo = do uid1 <- createUserPG pgInfo testUser3 uid2 <- createUserPG pgInfo testUser4 articleIds <- mapM (createArticlePG pgInfo)   [ testArticle2 uid1, testArticle3 uid1, testArticle4 uid1   , testArticle5 uid2, testArticle6 uid2 ] firstArticles <- runClientM (fetchArticlesByAuthorClient uid1) clientEnv secondArticles <- runClientM (fetchArticlesByAuthorClient uid2) clientEnv case (firstArticles, secondArticles) of   (Right as1, Right as2) -> return (entityVal <$> as1, entityVal <$> as2, uid1, uid2, articleIds)   _ -> error "Spec 5 failed!" Now we can write the assertion itself, testing that the articles returned are what we expect. spec5 :: SpecWith ([Article], [Article], Int64, Int64, [Int64])spec5 = describe "When fetching articles by author ID" $ do it "Fetching by the first author should return 3 articles" $ \(firstArticles, _, uid1, _, _) ->   firstArticles `shouldBe` [testArticle2 uid1, testArticle3 uid1, testArticle4 uid1] it "Fetching by the second author should return 2 articles" $ \(_, secondArticles, _, uid2, _) ->   secondArticles `shouldBe` [testArticle5 uid2, testArticle6 uid2] We would then follow that up with a similar after hook. The final test will follow a similar pattern. Only this time, we’ll be checking the combinations of users and articles. We’ll also make sure to include 12 different articles to test that the API limits results to 10. beforeHook6 :: ClientEnv -> PGInfo -> IO ([(User, Article)], Int64, Int64, [Int64])beforeHook6 clientEnv pgInfo = do uid1 <- createUserPG pgInfo testUser5 uid2 <- createUserPG pgInfo testUser6 articleIds <- mapM (createArticlePG pgInfo)   [ testArticle7 uid1, testArticle8 uid1, testArticle9 uid1, testArticle10 uid2   , testArticle11 uid2, testArticle12 uid1, testArticle13 uid2, testArticle14 uid2   , testArticle15 uid2, testArticle16 uid1, testArticle17 uid1, testArticle18 uid2   ] recentArticles <- runClientM fetchRecentArticlesClient clientEnv case recentArticles of   Right as -> return (entityValTuple <$> as, uid1, uid2, articleIds)   _ -> error "Spec 6 failed!" where   entityValTuple (Entity _ u, Entity _ a) = (u, a) Our spec will check that the list of 10 articles we get back matches our expectations. Then, as always, we remove the entities from our database. Now we call these tests with our other tests, with small wrappers to call the hooks: main :: IO ()main = do ... hspec $ before (beforeHook4 clientEnv pgInfo) $ after (afterHook4 pgInfo redisInfo) $ spec4 hspec $ before (beforeHook5 clientEnv pgInfo) $ after (afterHook5 pgInfo redisInfo) $ spec5 hspec $ before (beforeHook6 clientEnv pgInfo) $ after (afterHook6 pgInfo redisInfo) $ spec6 And now we’re done! The tests pass! …After creating and fetching an article The fetch call should return a result The article should be in Postgres

Finished in 0.1698 seconds2 examples, 0 failures

When fetching articles by author ID Fetching by the first author should return 3 articles Fetching by the second author should return 2 articles

Finished in 0.4944 seconds2 examples, 0 failures

When fetching recent articles Should fetch exactly the 10 most recent articles Conclusion This completes our overview of useful production libraries. Over these articles, we’ve constructed a small web API from scratch. We’ve seen some awesome abstractions that let us deal with only the most important pieces of the project. Both Persistent and Servant generated a lot of extra boilerplate for us. This article showed the power of the Esqueleto library in allowing us to do type-safe joins. We also saw an end-to-end process of adding a new type and endpoints to our API. In the coming weeks, we’ll be dealing with some more issues that can arise when building these kinds of systems. In particular, we’ll see how we can use alternative monads on top of Servant. Doing this can present certain issues that we’ll explore. We’ll culminate by exploring the different approaches to encapsulating effects. Be sure to check out our ! It’ll show you how to use Stack, so you can incorproate all the libraries from this series! Haskell Stack mini-course! If you’re new to Haskell and not ready for that yet, take a look at our and get going! Getting Started Checklist

Join the Club: Type-safe Joins with Esqueleto

In the last few articles, we’ve combined several useful Haskell libraries to make a small web app. We used to create a schema with automatic migrations for our database. Then we used to expose this database as an API through a couple simple queries. Finally, we used to act as a cache so that repeated requests for the same user happen faster. Persistent Servant Redis For our next step, we’ll tackle a thorny issue: testing. How do we test a system that has so many moving parts? There are a couple different general approaches we could take. On one end of the spectrum we could mock out most of our API calls and services. This helps gives our testing deterministic behavior. This is desirable since we would want to tie deployments to test results. But we also want to be faithfully testing our API. So on the other end, there’s the approach we’ll try in this article. We’ll set up functions to run our API and other services, and then use before and after hooks to use them. Creating Client Functions for Our API Calling our API from our tests means we’ll want a way to make API calls programmatically. We can do this with amazing ease with a Servant API by using the library. This library has one main function: . This function takes a proxy for our API and generates programmatic client functions. Let's remember our basic endpoint types (after resolving connection information parameters): servant-client client fetchUsersHandler :: Int64 -> Handler UsercreateUserHandler :: User -> Handler Int64 We’d like to be able to call these API’s with functions that use the same parameters. Those types might look something like this: fetchUserClient :: Int64 -> m UsercreateUserClient :: User -> m Int64 Where is some monad. And in this case, the library provides such a monad, . So let’s re-write these type signatures, but leave them seemingly unimplemented: m ServantClient ClientM fetchUserClient :: Int64 -> ClientM UsercreateUserClient :: User -> ClientM Int64 Now we’ll construct a pattern match that combines these function names with the operator. As always, we need to make sure we do this in the same order as the original API type. Then we’ll set this pattern to be the result of calling on a proxy for our API: :<|> client fetchUserClient :: Int64 -> ClientM UsercreateUserClient :: User -> ClientM Int64(fetchUserClient :<|> createUserClient) = client (Proxy :: Proxy UsersAPI) And that’s it! The Servant library fills in the details for us and implements these functions! We’ll see how we can actually call these functions later in the article. Setting Up the Tests We’d like to get to the business of deciding on our test cases and writing them. But first we need to make sure that our tests have a proper environment. This means 3 things. First we need to fetch the connection information for our data stores and API. This means the , the , and the we’ll use to call the client functions we wrote. Second, we need to actually migrate our database so it has the proper tables. Third, we need to make sure our server is actually running. Let’s start with the connection information, as this is easy: PGInfo RedisInfo ClientEnv import Database (fetchPostgresConnection, fetchRedisConnection)

...

setupTests = do  pgInfo <- fetchPostgresConnection  redisInfo <- fetchRedisConnection  ... Now to create our client environment, we’ll need two main things. We’ll need a manager for the network connections and the base URL for the API. Since we’re running the API locally, we’ll use a localhost URL. The default manager from the library will work fine for us: Network import Network.HTTP.Client (newManager)import Network.HTTP.Client.TLS (tlsManagerSettings)import Servant.Client (ClientEnv(..))

main = do  pgInfo <- fetchPostgresConnection  redisInfo <- fetchRedisConnection  mgr <- newManager tlsManagerSettings  baseUrl <- parseBaseUrl "http://127.0.0.1:8000"  let clientEnv = ClientEnv mgr baseUrl Now we can run our migration, which will ensure that our table exists: users import Schema (migrateAll)

main = do  pgInfo <- fetchPostgresConnection  ...  runStdoutLoggingT $ withPostgresqlConn pgInfo $ \dbConn ->    runReaderT (runMigrationSilent migrateAll) dbConn Last of all, we’ll start our server with from our API module. We’ll fork this off to a separate thread, as otherwise it will block the test thread! We’ll wait for a second afterward to make sure it actually loads before the tests run (there are less hacky ways to do this of course). But then we’ll return all the important information we need, and we're done with test setup: runServer main :: IO (PGInfo, RedisInfo, ClientEnv, ThreadID)main = do  pgInfo <- fetchPostgresConnection  redisInfo <- fetchRedisConnection  mgr <- newManager tlsManagerSettings  baseUrl <- parseBaseUrl "http://127.0.0.1:8000"  let clientEnv = ClientEnv mgr baseUrl  runStdoutLoggingT $ withPostgresqlConn pgInfo $ \dbConn ->    runReaderT (runMigrationSilent migrateAll) dbConn  threadId <- forkIO runServer  threadDelay 1000000  return (pgInfo, redisInfo, clientEnv, serverThreadId) Organizing our 3 Test Cases Now that we’re all set up, we can decide on our test cases. We’ll look at 3. First, if we have an empty database and we fetch a user by some arbitrary ID, we’ll expect an error. Further, we should expect that the user does not exist in the database or in the cache, even after calling . fetch In our second test case, we’ll look at the effects of calling the create endpoint. We’ll save the key we get from this endpoint. Then we’ll verify that this user exists in the database, but NOT in the cache. Finally, our third case will insert the user with the create endpoint and then fetch the user. We’ll expect at the end of this that in fact the user exists in both the database AND the cache. We organize each of our tests into up to three parts: the “before hook”, the test assertions, and the “after hook”. A “before hook” is some IO code we’ll run that will return particular results to our test assertion. We want to make sure it’s done running BEFORE any test assertions. This way, there’s no interleaving of effects between our test output and the API calls. Each before hook will first make the API calls we want. Then they’ll investigate our different databases and determine if certain users exist. We also want our tests to be database-neutral. That is, the database and cache should be in the same state after the test as they were before. So we’ll also have “after hooks” that run after our tests have finished (if we’ve actually created anything). The after hooks will delete any new entries. This means our before hooks also have to pass the keys for any database entities they create. This way the after hooks know what to delete. Last of course, we actually need the testing code that assertions about the results. These will be pretty straightforward as we’ll see below. Test #1 For our first test, we’ll start by making a client call to our API. We use combined with our and the function. Next, we’ll determine that the call in fact returns an error as it should. Then we’ll add two more lines checking if there’s an entry with the arbitrary ID in our database and our cache. Finally, we return all three boolean values: runClientM clientEnv fetchUserClient beforeHook1 :: ClientEnv -> PGInfo -> RedisInfo -> IO (Bool, Bool, Bool)beforeHook1 clientEnv pgInfo redisInfo = do  callResult <- runClientM (fetchUserClient 1) clientEnv  let throwsError = isLeft (callResult)  inPG <- isJust <$> fetchUserPG pgInfo 1  inRedis <- isJust <$> fetchUserRedis redisInfo 1  return (throwsError, inPG, inRedis) Now we’ll write our assertion. Since we’re using a before hook returning three booleans, the type of our will be . Each assertion will take this boolean tuple as a parameter, though we’ll only use one for each line. Spec SpecWith (Bool, Bool, Bool) it spec1 :: SpecWith (Bool, Bool, Bool)spec1 = describe "After fetching on an empty database" $ do  it "The fetch call should throw an error" $ \(throwsError, _, _) -> throwsError `shouldBe` True  it "There should be no user in Postgres" $ \(_, inPG, _) -> inPG `shouldBe` False  it "There should be no user in Redis" $ \(_, _, inRedis) -> inRedis `shouldBe` False And that’s all we need for the first test! We don’t need an after hook since it doesn’t add anything to our database. Tests 2 and 3 Now that we’re a little more familiar with how this code works, let’s take a quick look at the next before hook. This time we’ll first try creating our user. If this fails for whatever reason, we’ll throw an error and stop the tests. Then we can use the key to check out if the user exists in our database and Redis. We return the boolean values and the key. beforeHook2 :: ClientEnv -> PGInfo -> RedisInfo -> IO (Bool, Bool, Int64)beforeHook2 clientEnv pgInfo redisInfo = do  userKeyEither <- runClientM (createUserClient testUser) clientEnv  case userKeyEither of    Left _ -> error "DB call failed on spec 2!"    Right userKey -> do       inPG <- isJust <$> fetchUserPG pgInfo userKey      inRedis <- isJust <$> fetchUserRedis redisInfo userKey      return (inPG, inRedis, userKey) Now our spec will look similar. This time we expect to find a user in Postgres, but not in Redis. spec2 :: SpecWith (Bool, Bool, Int64)spec2 = describe "After creating the user but not fetching" $ do  it "There should be a user in Postgres" $ \(inPG, _, _) -> inPG `shouldBe` True  it "There should be no user in Redis" $ \(_, inRedis, _) -> inRedis `shouldBe` False Now we need to add the after hook, which will delete the user from the database and cache. Of course, we expect the user won’t exist in the cache, but we include this since we’ll need it in the final example: afterHook :: PGInfo -> RedisInfo -> (Bool, Bool, Int64) -> IO ()afterHook pgInfo redisInfo (_, _, key) = do  deleteUserCache redisInfo key  deleteUserPG pgInfo key Last, we’ll write one more test case. This will mimic the previous case, except we’ll throw in a call to in between. As a result, we expect the user to be in both Postgres and Redis: fetch beforeHook3 :: ClientEnv -> PGInfo -> RedisInfo -> IO (Bool, Bool, Int64)beforeHook3 clientEnv pgInfo redisInfo = do  userKeyEither <- runClientM (createUserClient testUser) clientEnv  case userKeyEither of    Left _ -> error "DB call failed on spec 3!"    Right userKey -> do       _ <- runClientM (fetchUserClient userKey) clientEnv       inPG <- isJust <$> fetchUserPG pgInfo userKey      inRedis <- isJust <$> fetchUserRedis redisInfo userKey      return (inPG, inRedis, userKey)

spec3 :: SpecWith (Bool, Bool, Int64)spec3 = describe "After creating the user and fetching" $ do  it "There should be a user in Postgres" $ \(inPG, _, _) -> inPG `shouldBe` True  it "There should be a user in Redis" $ \(_, inRedis, _) -> inRedis `shouldBe` True And it will use the same after hook as case 2, so we’re done! Hooking in and running the tests The last step is to glue all our pieces together with , , and . Here’s our main function, which also kills the thread running the server once it’s done: hspec before after main :: IO ()main = do  (pgInfo, redisInfo, clientEnv, tid) <- setupTests  hspec $ before (beforeHook1 clientEnv pgInfo redisInfo) spec1  hspec $ before (beforeHook2 clientEnv pgInfo redisInfo) $ after (afterHook pgInfo redisInfo) $ spec2  hspec $ before (beforeHook3 clientEnv pgInfo redisInfo) $ after (afterHook pgInfo redisInfo) $ spec3  killThread tid   return () And now our tests should pass! After fetching on an empty database  The fetch call should throw an error  There should be no user in Postgres  There should be no user in Redis

Finished in 0.0410 seconds3 examples, 0 failures

After creating the user but not fetching  There should be a user in Postgres  There should be no user in Redis

Finished in 0.0585 seconds2 examples, 0 failures

After creating the user and fetching  There should be a user in Postgres  There should be a user in Redis

Finished in 0.0813 seconds2 examples, 0 failures Using Docker So when I say, “the tests pass”, they now work on my system. But if you were to clone the code as is and try to run them, you would get failures. The tests depend on Postgres and Redis, so if you don’t have them running, they fail! It is quite annoying to have your tests depend on these outside services. This is the weakness of devising our tests as we have. It increases the on-boarding time for anyone coming into your codebase. The new person has to figure out which things they need to run, install them, and so on. So how do we fix this? One answer is by using . Docker allows you to create containers that have particular services running within them. This spares you from worrying about the details of setting up the services on your local machine. Even more important, you can deploy a docker image to your remote environments. So develop and prod will match your local system. To setup this process, we’ll create a description of the services we want running on our Docker container. We do this with a file. Here’s what ours looks like: Docker docker-compose version: '2'

services:  postgres:    image: postgres:9.6    container_name: prod-haskell-series-postgres    ports:      - "5432:5432"

  redis:    image: redis:4.0    container_name: prod-haskell-series-redis    ports:      - "6379:6379" Then, you can start these services for your Docker machines with . Granted, you do have to install and run Docker. But if you have several different services, this is a much easier on-boarding process. Better yet, the "compose" file ensures everyone uses the same versions of these services. docker-compose up Even with this container running, the tests will still fail! That’s because you also need the tests themselves to be running on your Docker cluster. But with Stack, this is easy! We’ll add the following flag to our file: stack.yaml docker:  enable: true Now, whenever you build and test your program, you will do so on Docker. The first time you do this, Docker will need to set everything up on the container. This means it will have to download Stack and ALL the different packages you use. So the first run will take a while. But subsequent runs will be normal. So after all that finishes, NOW the tests should work! Conclusion Testing integrated systems is hard. We can try mocking out the behavior of external services. But this can lead to a test representation of our program that isn’t faithful to the production system. But using the and hooks from Hspec is a great way make sure all your external events happen first. Then you can pass those results to simpler test assertions. before after When it comes time to run your system, it helps if you can bring up all your external services with one command! Docker allows you to do this by listing the different services in the file. Then, Stack makes it easy to run your program and tests on a docker container, so you can use the services! docker-compose Stack is the key to all this integration. If you’ve never used Stack before, you should check out our . It will teach you all the basics of organizing a Haskell project using Stack. free mini-course If this is your first exposure to Haskell, I’ve hopefully convinced of some of its awesome possibilities! Take a look at our and get learning! Getting Started Checklist

Tangled Webs: Testing an Integrated System

We all know that when Grand Theft Auto 6 comes out, it will instantly be one of the best-selling games of all time. The hype for this game is unprecedented and has been brewing for years. GTA 5 came out 11 years ago, and it was an instant success. It has sold 200 million copies, making it the second best-selling game of all time (only behind the juggernaut that is Minecraft), and has made over $8.5 billion in revenue. So, of course, the entire world is expecting bigger and better things for its sequel. And it looks like developer Rocksteady Games plans to deliver. GET PUBLISHED ON HACKERNOON.
Join a global network of  45,000+ published devs, builders, founders, makers, VCs, hodlrs, and  hackers. Start submitting your tech stories and tutorials to get  published FREE on HackerNoon — no pop-ups, no paywalls. According to a Rocksteady Games leak, the total budget of GTA 6 will be roughly $2 billion. No, I did not mistype a million. B for a billion. It’s important to note, however, that the budget encompasses everything: the creation of the base game, DLC, updates, marketing, etc. Even still, that’s an unbelievable amount of money. But if GTA 6 is just as much a raging success as its predecessor, the risk will definitely be worth the reward. And that brings up an interesting question: how does GTA 6’s budget compare to the other most expensive games ever made? Well, in order to see how it stacks up to the competition, I present to you the 5 most expensive games ever developed. P.S. We’ll only be looking at game budgets that have been confirmed. There are a lot of rumors and speculation about other games’ budgets, but those were never verified (Red Dead Redemption 2, Concord, etc.). That can be a separate list later 👀 5 Most Expensive Games Ever Developed 5. Marvel’s Spider-Man 3 Cyberpunk 2077 3. Monopoly Go! 2. Star Citizen 1. Genshin Impact 5. Marvel’s Spider-Man 3 - $385 Million In 2023, developer Insomniac Games had a giant leak, and when I say giant I mean GIANT. Polygon.com reported that the leak was 1.67 terabytes and contained 1.3 million files. It’s very unfortunate for Insomniac Games, but it did allow outsiders like us to get a peek at some of the technical/business aspects of gaming that we would never see. For example, a lot of the budgets for their games, including for their upcoming titles. And one of those titles is Marvel’s Spider-Man 3. According to the leak, Marvel’s Spider-Man 3’s budget is $385 million. This is a $70 million increase compared to Spider-Man 2’s budget. And who knows? Spider-Man 4 might hit the $400 million mark. 4. Cyberpunk 2077 - $441 Million Cyberpunk 2077 was a very ambitious game, even for revered developers CD Projekt Red, and the budget reflects that. In total, the budget of Cyberpunk 2077 was around $440 million, and what’s interesting is that, unlike other games, we can see exactly where that money was allocated. The budget for the base game was around $316 million. That right there already makes it one of the most expensive games ever made, but it didn’t end there. Cyberpunk 2077 was riddled with bugs at launch, and a lot of features that were promised were missing. Fixing all of that and making sure the game was updated to make it as good as possible wasn’t cheap. On top of that, the game also had a DLC expansion. In total, the post-launch budget was around $125 million, according to IGN. Put those 2 numbers together, and you get approximately $441 million. 3. Monopoly Go! - $500 Million Maybe it’s because I’m not too into mobile gaming, but I was completely blindsided when I found out that Monopoly Go! was the third most expensive game ever made, with a budget of $500 million. I know that Monopoly is iconic, but to me, it seems like a huge risk to put that much money into a Monopoly mobile game. Shows how much I know because the story gets even crazier. That $500 million isn’t even including the development cost. No, that $500 million is purely on marketing alone. That’s right, Monopoly Go! had a higher budget for marketing than the entire budget for Cyberpunk 2077 and Marvel’s Spider-Man 3. But the most wild part is that it paid off. The CEO confirmed that in the first 10 months, the game garnered $2 billion in revenue. I will never doubt Mr. Monopoly ever again. 2. Star Citizen - $700 Million Oh, Star Citizen. I was hearing about the development of this game when I was still in high school. One high school graduation, one college graduation, and one pandemic later, I’m still hearing news about the development of this game. According to IGN, Star Citizen’s budget officially surpassed the $700 million mark earlier this year. Although the game has been in development for over a decade, it’s not like it’s unaccessible or unplayable. You can go ahead and play the Alpha build of it right now. And a lot of people seem to enjoy it which, good for them, I say. However, it remains to be seen when it will officially be released and what the total budget will be when that day comes. 1. Genshin Impact - $700 Million It takes a lot of money to create a game, but as seen with Cyberpunk 2077, it also takes a lot of money to update it post-launch. But what happens when it’s a live-service game that has to continuously add new content and updates? Well, that means that the costs keep on coming. And that’s what’s happening with Genshin Impact. It’s been reported that the cost of just making Genshin Impact was $100 million. But that was just a drop in the bucket. miHoYo President Cai Hayou stated that it costs around $200 million a year to keep the new content coming. Although that might seem like a lot of money (because it is), it’s nothing compared to the revenue they’re bringing in. Earlier this month, it was announced that Genshin Impact reached $5 billion in revenue in China. With the game still going strong, it’ll be interesting to see what its total revenue will be when it’s all said and done. Conclusion What’s interesting to note is that if you combine the budgets of Star Citizen, Genshin Impact, and Monopoly Go!, it falls short of GTA 6’s budget by $100 million. I mean, really think about that. The 3 most expensive games ever released and their budgets still don’t touch the budget of GTA 6. If that’s not a testament to the high expectations put on the shoulders of the game, I don’t know what is. Only time will tell if it will reach those expectations. Who am I kidding? It definitely will. More in Gaming: 1. 5 Console-Level Games That You Can Play on Your iPhone Right Now 5 Gaming Franchises That Need to Return
Top 10 Greatest Games of All Time According to the Hackernoon Team Feature image source We all know that when Grand Theft Auto 6 comes out, it will instantly be one of the best-selling games of all time. The hype for this game is unprecedented and has been brewing for years. GTA 5 came out 11 years ago, and it was an instant success. It has sold 200 million copies , making it the second best-selling game of all time (only behind the juggernaut that is Minecraft), and has made over $8.5 billion in revenue . So, of course, the entire world is expecting bigger and better things for its sequel. 200 million copies $8.5 billion in revenue And it looks like developer Rocksteady Games plans to deliver. GET PUBLISHED ON HACKERNOON. Join a global network of  45,000+ published devs, builders, founders, makers, VCs, hodlrs, and  hackers. Start submitting your tech stories and tutorials to get  published FREE on HackerNoon — no pop-ups, no paywalls. GET PUBLISHED ON HACKERNOON. GET PUBLISHED ON HACKERNOON . GET PUBLISHED ON HACKERNOON Join a global network of  45,000+ published devs, builders, founders, makers, VCs, hodlrs, and  hackers. Start submitting your tech stories and tutorials to get  published FREE on HackerNoon — no pop-ups, no paywalls. Join a global network of  45,000+ published devs, builders, founders, makers, VCs, hodlrs, and  hackers. Start submitting your tech stories and tutorials to get  published FREE on HackerNoon — no pop-ups, no paywalls. According to a Rocksteady Games leak , the total budget of GTA 6 will be roughly $2 billion. No, I did not mistype a million. B for a billion. It’s important to note, however, that the budget encompasses everything: the creation of the base game, DLC, updates, marketing, etc. Even still, that’s an unbelievable amount of money. But if GTA 6 is just as much a raging success as its predecessor, the risk will definitely be worth the reward. Rocksteady Games leak And that brings up an interesting question: how does GTA 6’s budget compare to the other most expensive games ever made? Well, in order to see how it stacks up to the competition, I present to you the 5 most expensive games ever developed. P.S. We’ll only be looking at game budgets that have been confirmed. There are a lot of rumors and speculation about other games’ budgets, but those were never verified (Red Dead Redemption 2, Concord, etc.). That can be a separate list later 👀 5 Most Expensive Games Ever Developed 5. Marvel’s Spider-Man 3 Cyberpunk 2077 Cyberpunk 2077 3. Monopoly Go! 2. Star Citizen 1. Genshin Impact 5. Marvel’s Spider-Man 3 - $385 Million In 2023, developer Insomniac Games had a giant leak, and when I say giant I mean GIANT. Polygon.com reported that the leak was 1.67 terabytes and contained 1.3 million files. It’s very unfortunate for Insomniac Games, but it did allow outsiders like us to get a peek at some of the technical/business aspects of gaming that we would never see. For example, a lot of the budgets for their games, including for their upcoming titles. Polygon.com And one of those titles is Marvel’s Spider-Man 3. According to the leak, Marvel’s Spider-Man 3’s budget is $385 million. This is a $70 million increase compared to Spider-Man 2’s budget. And who knows? Spider-Man 4 might hit the $400 million mark. 4. Cyberpunk 2077 - $441 Million Cyberpunk 2077 was a very ambitious game, even for revered developers CD Projekt Red, and the budget reflects that. In total, the budget of Cyberpunk 2077 was around $440 million, and what’s interesting is that, unlike other games, we can see exactly where that money was allocated. The budget for the base game was around $316 million . That right there already makes it one of the most expensive games ever made, but it didn’t end there. $316 million Cyberpunk 2077 was riddled with bugs at launch, and a lot of features that were promised were missing. Fixing all of that and making sure the game was updated to make it as good as possible wasn’t cheap. On top of that, the game also had a DLC expansion. In total, the post-launch budget was around $125 million, according to IGN . Put those 2 numbers together, and you get approximately $441 million. IGN 3. Monopoly Go! - $500 Million Maybe it’s because I’m not too into mobile gaming, but I was completely blindsided when I found out that Monopoly Go! was the third most expensive game ever made, with a budget of $500 million . I know that Monopoly is iconic, but to me, it seems like a huge risk to put that much money into a Monopoly mobile game. Shows how much I know because the story gets even crazier. $500 million That $500 million isn’t even including the development cost. No, that $500 million is purely on marketing alone. That’s right, Monopoly Go! had a higher budget for marketing than the entire budget for Cyberpunk 2077 and Marvel’s Spider-Man 3. But the most wild part is that it paid off. The CEO confirmed that in the first 10 months, the game garnered $2 billion in revenue . I will never doubt Mr. Monopoly ever again. $2 billion in revenue 2. Star Citizen - $700 Million Oh, Star Citizen. I was hearing about the development of this game when I was still in high school. One high school graduation, one college graduation, and one pandemic later, I’m still hearing news about the development of this game. According to IGN , Star Citizen’s budget officially surpassed the $700 million mark earlier this year. IGN Although the game has been in development for over a decade, it’s not like it’s unaccessible or unplayable. You can go ahead and play the Alpha build of it right now. And a lot of people seem to enjoy it which, good for them, I say. However, it remains to be seen when it will officially be released and what the total budget will be when that day comes. 1. Genshin Impact - $700 Million It takes a lot of money to create a game, but as seen with Cyberpunk 2077, it also takes a lot of money to update it post-launch. But what happens when it’s a live-service game that has to continuously add new content and updates? Well, that means that the costs keep on coming. And that’s what’s happening with Genshin Impact. It’s been reported that the cost of just making Genshin Impact was $100 million. But that was just a drop in the bucket. miHoYo President Cai Hayou stated that it costs around $200 million a year to keep the new content coming. stated Although that might seem like a lot of money (because it is), it’s nothing compared to the revenue they’re bringing in. Earlier this month, it was announced that Genshin Impact reached $5 billion in revenue in China. With the game still going strong, it’ll be interesting to see what its total revenue will be when it’s all said and done. $5 billion in revenue Conclusion What’s interesting to note is that if you combine the budgets of Star Citizen, Genshin Impact, and Monopoly Go!, it falls short of GTA 6’s budget by $100 million. I mean, really think about that. The 3 most expensive games ever released and their budgets still don’t touch the budget of GTA 6. If that’s not a testament to the high expectations put on the shoulders of the game, I don’t know what is. Only time will tell if it will reach those expectations. Who am I kidding? It definitely will. More in Gaming: 1. 5 Console-Level Games That You Can Play on Your iPhone Right Now 5 Console-Level Games That You Can Play on Your iPhone Right Now 5 Gaming Franchises That Need to Return Top 10 Greatest Games of All Time According to the Hackernoon Team 5 Gaming Franchises That Need to Return 5 Gaming Franchises That Need to Return Top 10 Greatest Games of All Time According to the Hackernoon Team Top 10 Greatest Games of All Time According to the Hackernoon Team Feature image source Feature image source source

Stories, News & Mentions about RUNCHISE

Runchise’s Stories on HackerNoon