I have a dataset with 200,000 entries with four columns (time_of_day, order_size, time_taken, shop_number). I need to build a model and predict time_taken using the other three variables. There are more than 30,000 shop ids.  My approach has been to: 1. Try to use one_hot_encoding to encode each shop no. However, this leads to very large number of columns in my dataset. 2. Build a separate model for each shop assuming all shops have different efficiencies independent of one another.  Any other approach will be appreciated? Also, what type of model should I use in my data? I have tried simple linear regression and bayesian regression till now.
What techniques are available for collapsing (or pooling) many categories to a few, for the purpose of using them as an input (predictor) in a statistical model?    Consider a variable like  (discipline chosen by an undergraduate student). It is unordered and categorical, but it can potentially have dozens of distinct levels. Let's say I want to use major as a predictor in a regression model.  Using these levels as-is for modeling leads to all sorts of issues because there are just so many. A lot of statistical precision would be thrown away to use them, and the results are hard to interpret. We're rarely interested in specific majors -- we're much more likely to be interested in broad categories (subgroups) of majors. But it isn't always clear how to divide up the levels into such higher-level categories, or even how many higher-level categories to use.  For typical data I would be happy to use factor analysis, matrix factorization, or a discrete latent modeling technique. But majors are mutually exclusive categories, so I'm hesitant to exploit their covariance for anything.  Furthermore I don't care about the major categories on their own. I care about producing higher-level categories that are coherent with respect to my regression outcome. In the binary outcome case, that suggests to me something like linear discriminant analysis (LDA) to generate higher-level categories that maximize discriminative performance. But LDA is a limited technique and that feels like dirty data dredging to me. Moreover any continuous solution will be hard to interpret.  Meanwhile something based on covariances, like multiple correspondence analysis (MCA), seems suspect to me in this case because of the inherent dependence among mutually exclusive dummy variables -- they're better suited for studying multiple categorical variables, rather than multiple categories of the same variable.  edit: to be clear, this is about collapsing categories (not selecting them), and the categories are predictors or independent variables. In hindsight, this problem seems like an appropriate time to "regularize 'em all and let God sort 'em out". Glad to see this question is interesting to so many people!
is_active_sidebar() Always Returns False Comment character counter does not trim, while the server does How to change the title of ToC? I need to change default title of ToC. Is it possible with titlesec package? Here is my MWE:  \documentclass{book}  \usepackage{titlesec} \titleformat{\chapter}{\normalfont\Large\scshape}{\thechapter}{1em}{} \vspace{6pt}]  \usepackage{lipsum}   %-------------------------------------------------------------------------  \begin{document}   \newpage \thispagestyle{empty} \tableofcontents \thispagestyle{empty} \newpage \thispagestyle{empty}  \setcounter{footnote} {0}\chapter*{{\fontsize{11}{14}\selectfont\normalfont{Author1} \\ {\scshape\Large Title}}} \addcontentsline{toc}{chapter}{\normalsize\normalfont{Author1} \\ \normalfont\scshape{Title}} \thispagestyle{empty} \Author{Author1} \Title{Title} \lipsum \newpage \thispagestyle{empty}    \end{document} Using ExactTarget DEManager to update record? Is there a way to use ExactTarget DEManager to update a data extension record instead of only add? I have a data extension that adds subscribers to specified lists (each list is its own boolean column). If a user goes to my DEManager form and the subscriber already exists, the form will return the error url ("subscriber already exists" or something similar). Is there any way instead for the form to update the existing subscriber record instead of it returning an error? If so, is there a way to use DEManager to first check if a record exists, and if it doesn't then add, otherwise update?  Reference: the DEManager section at   &lt;form action="http://cl.exct.net/DEManager.aspx" name="subscribeForm" method="post"&gt;         &lt;input type="hidden" name="_clientID" value="12345" /&gt;         &lt;input type="hidden" name="_deExternalKey" value="WebCollectSubs" /&gt;         &lt;input type="hidden" name="_action" value="add" /&gt;         &lt;input type="hidden" name="_returnXML" value="0" /&gt;         &lt;input type="hidden" name="_successURL" value="http://example.com/Success" /&gt;         &lt;input type="hidden" name="_errorURL" value="http://example.com/Failed" /&gt;         Email: &lt;input type="text" name="Email"&gt;&lt;br /&gt;         First Name: &lt;input type="text" name="FirstName"&gt;&lt;br /&gt;         Last Name: &lt;input type="text" name="LastName"&gt;&lt;br /&gt;         &lt;input type="submit" value="Submit"&gt;     &lt;/form&gt; I have a virtual machine running ubuntu and I am trying to host some files on ubuntu and map the drive in windows PC.  Following the steps , I am able to see (from windows) the "work" folder after typing in \ubuntu_ip. But when trying to access or map it with "svc_acct" from windows, I get "0x80070043 error the network name cannot be found". I guess I am not accessing correctly from windows.   SAMBA config below (svc_acct is an account set up under linux). When accessing the folder from windows, I get prompted for username/password. Not sure what credential to type in exactly.  samba config: [work] path = /home/svc_acct/work valid users = svc_acct available = yes read only = no browsable = yes public = yes writable = yes force user = svc_acct create mask = 0777 directory mask = 0777   UPDATE:  All I did to resolve this is by changing samba config's path from path = /home/svc_acct/work to path = /home/svc_acct following  post  (last sentence). Then no more prompt for password in windows when I try to click into the folder. Thanks to @sancho.s for suggesting the link. Log of products to sum of logs If $\mathbf A$ and $\mathbf B$ are Hermitian, when does $\det(\mathbf A-\lambda\mathbf B)=0$ have only real roots? Let $\mathbf A, \mathbf B$ be Hermitian matrices of the same size. What is the characterization of $\mathbf A, \mathbf B$ such that $p(\lambda)=\det(\mathbf A-\lambda\mathbf B)=0$ has only real roots?   If $\mathbf B$ is positive definite (I corrected this), it is easy to see $p(\lambda)$ has only real roots. Are groups satisfying $g=g^{-1}$ for all $g \in G$ abelian? Is the following statement true or false:  If $G$ is a group with the property that $g=g^{-1}$ for all $g \in G$, then $G$ is abelian.   I believe it is false since I know that abelian or commutative property implies that every element in $G$ must have an inverse. Thus $g\cdot g^{-1}= g^{1}\cdot g = e$. I need someone to check my attempt. OS X updater claims installation is successful, but new OS is not installed How can we define convexity in one dimension? Definition of a convex set: A set $S\subset\mathbb R^n$ is said to be convex if for each $x_1, x_2\in S$ the line segment $\lambda x_1 + (1-\lambda)x_2$ for $\lambda\in (0,1)$ belongs to $S$. This says that all points on the line connecting two points in the set are in the set. My question is: how can we define convexity in one dimension? Problems with Groups in Traminer I'm having a problem using the group function in TraMineR. I have a data set that contains SPELL data, so multiple rows per case. I also have demographic data per case, at one row per case. I merge these together and end up with data that has a demographic covariate per row, so multiple rows per case. An example of this data would be:  id startmin stopmin activity educ4 4      1      20       work    HS 4      20     40       play    HS 8      1      15       sleep   College 8      15     40       work    College   I can make sequence data from this, but when I try to run a plot using the group command        seqiplot(atus.seq, group = atus.seqdata$educ4, border=NA,                 withlegend="right",  cex.plot=.55)   it tells me:       &gt;"Error: group must contain one value for each row in the sequence object".   I have gotten this to work with the example of the mvad data in the training manual, but I can't seem to get it to work with the groups, whether I link to the original demographic data, the merged data, or try to pass the covariates by seqformat and seqdef. Ideas? biblatex in a nutshell (for beginners) Is there a 'biblatex in a nutshell' guide out there? I'd like if someone explained to me the essentials on how to use biblatex (what lines I have to write in my document, which files I have to have, how many times and what I have to compile), so then I can go to  to customise it further. Lost access to sudo sudo: unable to resolve host   I think I need to edit the /etc/hosts file to make an entry for 127.0.1.1 PC name.  But how can I do that if I can't access sudo? It seems to have gone since I updated to 13.10.  I have tried nano, which still doesn't work. "Least Astonishment" and the Mutable Default Argument Distribution of the ratio of two i.i.d standard normals I am reading an article in which the author states that given two i.i.d random variables $X,Y\sim\mathcal{N}(0,1)$, we have $\mathbb{P}(\frac{X}{Y}\le t)=\mathbb{P}(\frac{X}{|Y|}\le t)$ since the random varaibles $\frac{X}{Y}$ and $\frac{X}{|Y|}$ are identically distributed by the symmetry of the standard normal distribution.  But I don't really understand why the random variables $\frac{X}{Y}$ and $\frac{X}{|Y|}$ are identically distributed. Can you provide a reason for why these two random variables are identically distributed? In a letter-class document that had no previous errors I am suddenly getting the error message:  "There's no line here to end" \opening{text}   I've used this file many times and today it starts giving this error. How do I troubleshoot this? Images instead of radio buttons or dropdown list? More Generating Functions problems Is it possible to add non-number characters on the left of a code block?  Specifically, I want to add ... to correspond with some source code (see image).    I am currently using commands to stop / start the line numbers. Here is the code:  \documentclass[12pt]{article}  \usepackage[a4paper, margin = 20mm, top = 30mm, bottom = 25mm]{geometry} \usepackage{xcolor, soul} \usepackage{listings}  \definecolor{codebackground}{rgb}{0.95, 0.95, 0.92}  \lstset {     frame            = tb, % hrule above and below     keepspaces       = true,     columns          = flexible,     basicstyle       = {\normalsize\ttfamily},     escapeinside     = {(*@}{@*)}, % for escaping     backgroundcolor  = \color{codebackground},     showstringspaces = false,     language         = C,     keywordstyle     = \color{blue},     stringstyle      = \color{red},     commentstyle     = \color{teal},     numbers          = left, % {none, left, right}     firstnumber      = 1,     numberstyle      = \scriptsize\color{black},     numbersep        = 5pt }  \let\origthelstnumber\thelstnumber \makeatletter % for stopping line numbers \newcommand*\stopnumber {     \lst@AddToHook{OnNewLine}     {         \let\thelstnumber\relax         \advance\c@lstnumber-\@ne\relax     } } % for starting line numbers \newcommand*\startnumber[1] {     \setcounter{lstnumber}{\numexpr#1-1\relax}     \lst@AddToHook{OnNewLine}     {         \let\thelstnumber\origthelstnumber         \refstepcounter{lstnumber}     } } \makeatother  \begin{document} \begin{lstlisting}[firstnumber = 456] int third_function(const int c) {     /* error checking */     if(c &lt; 0)     {         return;     } (*@ \stopnumber @*)      ...     (*@ \startnumber{512} @*)     /* other stuff */     int d = 4;     char* word = "third function"; (*@ \stopnumber @*)      ...     (*@ \startnumber{678} @*)     return (c * d); } \end{lstlisting} \end{document} Creating or referencing variables dynamically in Sass